2

Tieing Elements and Versions into Render Management

Gavin Greenwalt

July 22, 2009 01:22

None

Follow

I'm contemplating the following workflow for Shotgun and Renders.   Has anyone tried something similar or have any comments on shotgun which I should think about differently?

----

Instead of using Assets to track renders I'm thinking about using assets purely as source files (3D Scenes, Compositing Scripts/Workspaces, Edit Timelines, 3D Models, Textures, etc).  

Elements on the other hand would be plates and render passes.   Beauty, Spec, Data, Cleanplate etc....

Every time a render completes it would be tagged in the render management software with:
Sequence -> Shot  -> Element.

The render management software would create the proxy quicktime and thumbnail.  Create a new version for the element and link the file path of the render output to the new version.  

It would also create a link to the render's source asset version.  (BigSceneElement -> Beauty -> Version 1) LINKED TO (BigSceneFile -> Version 10).  This way an artist would know where the render came from.

In Nuke for example now you would read node which let you chooose:
Project: Big Project
Sequence: Seq_01
Shot: Shot_01
Element: Beauty
Version: version 1

If the render management software adds a version 2 (say linked to big scene asset version 11) then an alert would be pushed to the artist that an element has a newer version available.

---------

Is splitting up source files and renders as assets and elements respectively a sensible strategy or should I keep everything in assets?   Is this a good way to lay out Shotgun projects in general?  Is anyone else currently tracking versions via shotgun and have any comments they would be willing to share on how they approached it?  Any feedback would be appreciated.

• Facebook
• Twitter
• LinkedIn

13 comments

• 0
  
  Armando Ricalde September 11, 2010 00:30

  Hi Gavin

  More than a year ago you asked about this, but no one got interested, and now I'm thinking about just the same.
  How did you solve it? Does this workflow worked well for you?

  I think the place where the "published" versions (scenes, textures, renders, etc) go should be "read-only" for the users if not the relationship between the file system and SG could be broken very easily.

  As you said any feedback would be really appreciated. 

• 0
  
  Mike Romey September 28, 2010 16:34

  Hi Gavin, We have tied our render management to shotgun. We use Rush. The problem was the front end of rush. We ended up writing our own front end for rush. The front end allows users to review content in RV, publish QuickTimes and thumbnails to shotgun, mash up shotgun pages and notes, timecards, render logs and transfer between locations via aspera. We opted not to automatically publish versions and elements to shotgun since we required artists to qc content prior to publishing. Not all versions are intended to be reviewed. Our tools work for Mata, Nuke, After Effects, Lightwave, Mach Studio, Mental Ray and . One other interesting thing we do is generate a new custom render log entry where we upload statistics about the renders but also upload dependency reports generated at submission time of all the content used to render that scene. this includes textures, cache, references, etc etc etc. this has become very helpful for archiving and storage purging purposes. He some of these details help. happy to answer any other questions you might have. Romey Romey
• 0
  
  Armando Ricalde September 28, 2010 17:58

  Hi Mike, we use Rush too :)

  Do you ended up writing your own version of iRush? I was thinking in writing custom submission scripts as a start, so when the render successfully finishes publish a render log, stats and a Version for an Element to shotgun.

  Mmhhh, I think we still have too much work to do. Thanks

• 0
  
  Mike Romey September 28, 2010 18:07

  Yes, we wrote a more advanced version of iRush using python and qt with fast charting and stats. We also started by writing our own submitters. Our submitters do a pre Shotgun entry creation and a post Render shotgun entry update. All actions from Rush update those entries such as dumping or continuing. My advice would be to start with the submitter, then migrate to the manager. Pretty sure you can have rush run commands on existing jobs on the stack. So you could outfit Irish to help out. Romey
• 0
  
  Jonas Drehn December 17, 2011 23:23

  Hi Guys,

  I'm also looking into this now ... how to work with 3D renders, but also with RAW footage / plates. Essential I will be working on creating a entry for each plate shot on set - so we can keep track of these inside SG - I was thinking about putting them into elements, but I think it's important to be able to playback these elements in RV - which dosen't seem to be supported ? so would it be a better way - just to use assets for this, if yes, I would also need a version for each asset right ? which on plates is a not necessary - but makes sense for 3d renders.

   

  Love to hear where you guys put your plates and renders

• 0
  
  Diogo Girondi February 23, 2012 21:07

  Even though I never felt too sure about how I organized things in SG regarding that bit, what I did in the past worked and had it's advantages from certain standpoints. And it's probably what I will keep doing until I find a better way.

  I used Assets to track models, 3D projects, comp projects, matte painting and whatnots but not plates/footage. Versions linked to Shots were used for "final/output/review" renders of each shot that most likely would be sent to "DI/Grading". And Versions linked to Elements to manage anything that was either the source material or intermediary renders that were coming from 3D, MoGraph, or other departments that would have to pass trough the comp department.

  The reason to use Versions linked to Elements was simple. First I needed to keep the RV integration to review plates and whatnots, and using Elements directly would block me from having that. Secondly it allowed me to have iterations of work done to element inside a single entity under a single ID, which for me was easier to track during Comp stages since I could simply query the latest or active version of a given element. 

• 0
  
  Gavin Greenwalt February 23, 2012 21:14

  Agreed with Diogo, and to answer your more than a year old question Armando it has been working well and nothing has come up which would inspire me to do anything differently. 

• 0
  
  Kevin Sallee August 16, 2012 17:41

  Hi there!
  SG and Greg Ercolano from rush worked on a UI to submit this kind of stuff. I use shotgun_io and rushShotgun to publish versions. We normally publish our final/output/review Versions with a type "Progress", so I modified a little bit rushShotgun code so I can tag these particular renders (intermediary work) as elements. I also hacked shotgun_io and rushShotgun so it could fill in the field sg_uploaded_movie, that way it uploads the .mov to SG, creates a .mp4 and a filmstrip, and my thumbnail becomes a lot more interesting.
  I then redesigned the Review page and Assets and Shots pages so I can have a list of all Versions of type element at hand very quickly.
  
  Having done all that, I was discussing with my boss Armando, and he told me about this page of the forum. The thing is I think having these versions with type element is enough for our purposes, but seeing this conversation I see that the "standard" workflow would be to create an element (with naming convention Shot_Task_customPassSuffix (i.e. S010_Render_SSS), and then Versions linked to this element (i.e. S010_Render_SSS_v01). I already have the latter, but for the moment I don't really see the point of creating an element for this for v01, and then searching for the right element when I publish a new version of something.
  Also the thing is that shotgun_io was thought of for publishing Versions from what I saw, and hacking into that for all the element behaviour would be kind of difficult, I would have to add another sg instance in my code and do some python requests to SG that would slow down a little bit the process. Or maybe do a daemon to create the elements and link the Versions correctly when created.
  What I would really want to know is what is the advantage of having the Elements and Versions of type Element instead of only the latter.

   

• 0
  
  Gavin Greenwalt August 16, 2012 18:10

  The advantage of using 
  
  Shots\Elements\Versions is for navigation.
  
  If I'm an artist I want:    Project: ABC \ Shot: 0010 \ Element: BEAUTY \ Latest Version.
  If then down the road I'm in COMP and LIGHTING gets a revision I can see right in side of nuke that there is a new version of ABC_0010_BEAUTY that I need. 
  
  It just generally in my opinion works better into a Relational database philosophy where every piece of redundant information is in a separate table.    In this instance as an artist I'm looking for a specific render element--I want the system to tell me which version is the one I want. 
   

• 0
  
  Kevin Sallee August 16, 2012 18:21

  "It just generally in my opinion works better into a Relational database philosophy where every piece of redundant information is in a separate table"
  if it's redundant it's probably not in the same table....
  And again, I can understand the need of Element but not its absolute need, because what you're talking about can be done without the Element thing.
  For example in my review pages I can group Elements by task and sort them by name, that way I always get the latest version at the top.
  For comp (we work with nuke), read nodes can be aware that a new version has been published and update it without the need of an element. These versions are already tied by their name, their link to the shot/asset, and the task they're linked to. That's enough to separate versions without the element part, I think.
  
  I understand it can be more easily accessible and queried with the Element, but this would require a certain amount of extra work, and i'm just trying to see if it's really crucial to have them. 

• 0
  
  Kevin Sallee August 16, 2012 18:22

  "It just generally in my opinion works better into a Relational database philosophy where every piece of redundant information is in a separate table"
  if it's redundant it's probably not in the same table....
  And again, I can understand the need of Element but not its absolute need, because what you're talking about can be done without the Element thing.
  For example in my review pages I can group Elements by task and sort them by name, that way I always get the latest version at the top.
  For comp (we work with nuke), read nodes can be aware that a new version has been published and update it without the need of an element. These versions are already tied by their name, their link to the shot/asset, and the task they're linked to. That's enough to separate versions without the element part, I think.
  
  I understand it can be more easily accessible and queried with the Element, but this would require a certain amount of extra work, and i'm just trying to see if it's really crucial to have them. 

• 0
  
  Gavin Greenwalt August 16, 2012 18:36

  By redundant I mean shot name, element name etc. 
  
  If you have
  
  Table Shots: [shot GUID, shot Name]
  Table Elements: [elem GUID, elem Name, shot GUID]
  Table Version: [ver GUID, ver Name, elem GUID]
  
  And you call "version 3902139" you can work your way back up the tree and reconstruct from a version name such as "v001" -> Link -> "Element SSS" -> Link -> Shot 0010 -> Link -> Project ABC.     If somewhere down the line your project name were to change that change would dynamically and automatically flow down through to every version since it's linked to the project at the database level.    You aren't manually (and potentially in an error prone fashion embedding "SSS"  or "Project ABC" in 100 versions redundantly you're just linking to a single dynamic and editable entry. 
  
  So is it crucial? Probably not.  But I would argue it makes your data structure a lot more resilient to change.   You aren't trying to parse out its identity from an arguably unreliable version name.  If somehow an artist then creates a new version called ABC_0010_SSD_v005 on accident if you don't know what element it's *supposed* to be then a version name regEx might mistakenly believe that you have a new element named "SSD" instead of "SSS".   Keeping "SSS" as a unique element entry ensures that a small typo won't lose that version number. 
   

• 0
  
  Kevin Sallee August 16, 2012 18:44

  mm my version are dynamically linked to their respective project as they should be, so if you rename your project it automatically modifies my versions so from this standpoint i dont see any problems.
  And making mistakes in their versions shouldn't be that easy, since our submit tool uses a dropdown menu that offers them these kind of entries: ShotName_TaskName_suffix with suffix being all the kind of passes the artists told me they used (DIF,IBI,SSS,SSS_EPIDERMAL,etc) So the name is not prone to error i think.
  Also I would argue that from my point of view, if i created an element, I would use the information the user submits in the Submit UI, i.e. right now the entry i use to name the version would be used to check if an element with this name exists and create a new one if not, so if they got the name of the version wrong, i.e ABC_0010_SSD (they never enter the version number because it's done automatically), an element with the wrong name would be created and the versions with the right name and the wrong name would be split into two different elements which is also a hassle for repairing these kinds of mistakes. So as for the naming convention, we are kind of very strict to avoid any mistakes like this one.
  
  Any more arguments? I'm still not convinced :)