Setting up a {CustomEntity}/{Sequence}/{Shot} hierarchy

Hasiel Hassan Alvarez Zavala

July 28, 2015 16:21

None

I'm trying to use a custom entity in the context, but not really sure if I understand how the context_additional_entities hook works. I want to structure Shots folders like this:

Shots/{CustomEntity}/{Sequence}/{Shot}

But, the engine crash because the context don't have a CustomEntity. I tried to use the context_additional_entities "hook", populating the entity_fields_on_task with the field that points to a CustomEntity, but then, the engine crash because it can't assing values to {Shot}, {Sequence} and {Step}

Can you explain me this workflow ?

Date Votes

15 comments

0
Permanently deleted user July 28, 2015 19:07
Hello Hasiel!

Excellent question - thanks for posting!

Sorry, the context_additional_entities hook is a red herring - something historical that we have been looking at changing for quite some time now.

Setting up a Shots/{CustomEntity}/{Sequence}/{Shot} hierarchy should be (relatively) straight forward and you don't need to use the context_additional_entities. Let me outline the steps! For simplicity, let's assume that we are setting up a structure were we have episode / sequence / shot and the episode data is called CustomEntity01 under the hood.

1. Make sure you have CustomEntity01 enabled in the Shotgun Preferences.

2. Add a custom "episode" (sg_episode) field to Sequence which is a single entity link to episode, so that you can link a single episode to each sequence.

3. In the toolkit folder schema configuration, add in a new folder hierarchy level for episode. Link up your sequence level folder with the episode parent, as shown in the image below:

4. Now you should be able to create folders on disk with the desired episode/sequence/shot structure!

5. Next step is to make sure all the apps work. This is done by editing the templates.yml file, ensuring that all the paths contain the episode token. You do this by creating a key definition in the keys section:
```
CustomEntity01:
   type: str
```
Then go through and update all the relevant paths:
```
shot_root: sequences/{Sequence}/{Shot}/{Step} ->
shot_root: sequences/{CustomEntity01}/Sequence}/{Shot}/{Step}
```
Unfortunately, there is currently no support for saying "episode", but you have to use the system name for the entity type.

6. All done! Now you should be able to launch maya and start publishing etc, just like you would with your normal sequence/shot structure.

For more information and further steps, check out the following post: https://support.shotgunsoftware.com/entries/95444048 . Hope this helps! Do you think this will do the trick?

Thanks!
Manne
Comment actions Permalink
0

Rodrigo Rodriguez July 28, 2015 19:07

Hi!

So this fix populates the "Additional Entities" property in the context?
By the way, what if the query at episode.yml returns more than one entity?

I followed the steps and additional_entitites it's still empty. We are using the tank core version v0.14.45 and it keeps giving me the error:

Context Animation, Shot Sht001 can not determine value for fields ['PublishedFileType'] needed by template

Thanks!

Rodrigo

Comment actions Permalink
0

Permanently deleted user July 28, 2015 19:07

Hello!

What I am basically suggesting is to ignore additional_entities completely. You don't need to have that populated for the template system to work - it is essentially an older part of toolkit which we may deprecate at some point. Remember that the template system can for example resolve both a Sequence and a Shot from a context where the entity is simply "Shot" - this is because the Sequence is linked to the Shot and toolkit takes advantage of that relationship.

I think the error you are getting probably relates to something else - if you send your config across I can take a quick look!

> By the way, what if the query at episode.yml returns more than one entity?

The folder creation system does not currently support the notion of multi entity joins - basically, each object in the file system hierarchy must have a distinct parent - like a sequence belonging to a single project, a shot belonging to a single sequence etc. It is therefore for example not currently possible to add all the assets that are associated with a Shot as folders underneath the shot level (although that would be interesting!) - because an asset may potentially be associated with more than one shot - it then becomes complicated and ambiguous which folder you actually refer to!

We are very aware that some of the concepts around the context object can be a bit confusing an it has been on the refactoring list for quite some time now - we are hoping to reshuffle it a bit, hopefully ending up with something that is a lot clearer conceptually and more flexible.

Any questions, don't hesitate to reach out!

Manne

Comment actions Permalink
0

Rodrigo Rodriguez July 28, 2015 19:07

Hello Manne!

Let me explain our problem in more detail. I don't know if other studios have this kind of issue or if it's just us. It could be great if other people share their ideas on this.
We have something like 20-30 open projects per month (all of them with different sizes, reaches and lifetimes). By the force of evolving workflows and continuous development we end up with a templates.yml file with more than 600 lines. That means 20-30 files with 600 lines of configurations!. It turned into a management nightmare, with many templates being something like:

maya_shot_playblast_version: '@shot_publish/maya/review/{project_code}_{Shot}_{step_code}_v{version}_{artist}.mov'
nuke_shot_render_movie: '@shot_publish/nuke/review/{project_code}_{Shot}_{step_code}_v{version}_{artist}.mov'

We use those quicktimes as versions for revisions and want to keep all versions using the same file structure, since the only difference between those templates is the software name I thought that it could be nice if we have a {software} key or even a {ext} key, for work files (maybe you remember this from my other issues). With this we could use the same template for every app config that generates a version file or workfile or publish file or image sequence, etc. the problem is how do we solve those keys?. There is no problem inside our custom apps, but we also want to keep using your apps and there is no customizable hook to solve paths, so the info has to be inside the context, but where? and when?

I put extra fields in the PublishFileType entity to keep track of software and file extension by file type, by brute force I added the PublishFileType instance to the context's additional entities and it worked perfectly, at least for workfiles, publish files and snapshots (maybe for versions and sequences after some changes). My purpose is end up with a templates.yml like the one attached.

When? the file type can only be decided until the engine starts, I put the brute force to fill the context on the before_app_launch hook from the multi launch app but the engine doesn't keep the context's additional entities after opening a file. I also tried editing the engine before the app start, this work just as I wanted the problem is it means add custom code to every engine (not a god idea, unless you add a hook for that).

Why don't we use the solution from your example? Because the PublishFileType doesn't have a one to one relation with any other entity in the context.
I'll continue cleaning our mess, as I said maybe this is very specific to us, thanks for your attention, any advice will be welcome.

Rodrigo

Comment actions Permalink
0
Permanently deleted user July 28, 2015 19:07
Hey there Rodrigo!

Thanks for the great detailed feedback! Not only does it help us understand what you are trying to do, but I think it is a really interesting read for all of the community!

One thought that immediately sprung to mind was if we added a feature which lets you resolve a key based on the current engine. What if you could define the following key in your templates.yml?
```
keys:
  curr_engine:
    type: str
    default: {current_engine}

templates:
  shot_playblast_version: '@shot_publish/{curr_engine}/review/{project_code}_{Shot}_{step_code}_v{version}_{artist}.mov'
```
This key would resolve to tk-nuke, tk-maya etc. Do you think that would be a good solution - or the beginnings of a solution? I suspect this is a relatively simple addition for us!

Thanks!
Manne
Comment actions Permalink
0
Rodrigo Rodriguez July 28, 2015 19:07
Hi Manne!

That could be a great start!.

Over this days I had an idea, and I would like to know your opinion. What if we could pre-eval some keys on the templates.yml file? for example:
```
shot_version: '@shot_publish/{software}/review/{project_code}_{Shot}_{step_code}_v{version}_{artist}.{ext}'

shot_maya_version: ' @shot_version(software: maya, ext: mov)'
```
I was looking at the yml specs for something like this but I didn't find anything. This could lead to the problem of two templates matching the same path, but maybe we could have the parent templates in other area. It's just an idea.

Regards!

Rodrigo
Comment actions Permalink
0

Permanently deleted user July 28, 2015 19:07

That's a really interesting idea!

Would you prefer the shot_maya_version: ' @shot_version(software: maya, ext: mov)' method to the idea of having a {current_engine} default value? With the former, you would still have a lot of lines in your config, but it would be a lot more readable! With the latter, you would have fewer templates.

Comment actions Permalink
0

Rodrigo Rodriguez July 28, 2015 19:07

Which one do you think is better? easier/faster to implement? useful?

Thanks!

Comment actions Permalink
0
Permanently deleted user July 28, 2015 19:07
I like the idea of the {current_engine} auto resolve default, but I think there needs to be more configurability! Maybe more like an enumeration:
```
keys:
  curr_engine:
    type: str
    default: [current_engine, {tk-maya:maya, tk-nuke:nuke, default:generic}]
```
This would allow you to configure the exact values you want, but drive it from a procedural value (e.g. the engine name). The syntax needs to be refined obviously - I'll run it by the tookit group this week and will see what they think!
Comment actions Permalink
0

Rodrigo Rodriguez July 28, 2015 19:07

Great!

Comment actions Permalink
0
Permanently deleted user July 28, 2015 19:08
Hello Hasiel!

I brainstormed this with the engineering team and we came up with two approaches we would like to run by you. Both of them use the extended templates definition syntax we already use for multi root configs.

The first one is similar to what you are suggesting and would look like this in templates.yml:
```
shot_{profile}_work: 
  definition: sequences/{Sequence}/{Shot}/{Step}/{software}/work/{Shot}.v{version}.{ext}
  profiles:
    maya: {software: Maya, ext: [ma|mb]}
    nuke: {software: Nuke, ext: nk}
```
In the environment settings for an app, you would then either refer to this template as shot_maya_work or shot_nuke_work and its {engine} and {ext} fields would be replaced with values according to the recipe defined. Toolkit would look for the special {profile} keyword in the template name and automatically expand this profile into shot_maya_work and shot_nuke_work as it is loading the templates. As you are introspecting the API, you would never see the shot_{profile}_work template.This would allow you to set up a relatively lean set of template entries and you avoid having to have a shot_maya_work, shot_nuke_work, shot_houdini_work etc.

The second approach is more explicit:
```
shot_generic_work: sequences/{Sequence}/{Shot}/{Step}/{software}/work/{Shot}.v{version}.{ext}

shot_maya_work: 
  definition: @shot_generic_work
  replacements:
    engine: maya
    ext: mov

shot_nuke_work: 
  definition: @shot_generic_work
  replacements:
    engine: nuke
    ext: qt
```
The approach is very similar in that it effectively does a pre-processing step, but it is more explicit in that what you see in the template file matches more exactly what is loaded into the tk.templates dictionary and you have available at runtime. So by looking at the templates file you can more easily overview the total list of templates - however that of course also means that the total list of templates will be longer.

Curious to hear your thoughts!

Manne
Comment actions Permalink
0

Rodrigo Rodriguez July 28, 2015 19:08

Hi Manne!

I think the first approach would be a lot easier to manage! Do you think that this approach could be transparent for your apps?

Thanks!

Rodrigo

Comment actions Permalink
0

Permanently deleted user July 28, 2015 19:08

Hello!

This approach should be compatible with all our existing apps, which is nice! We will be discussing it tomorrow in one of our engineering catchups - will get back with some more info after that… :)

Comment actions Permalink
0

Permanently deleted user July 28, 2015 19:08

Hello Rodrigo!

It's been quite a while since this thread was active but I wanted to do a bit of a followup. The idea of a preprocessor and more easy and powerful ways to manipulate the configuration to make things clearer and easier to maintain has made it in as one of the main themes for our more large scale engineering refactors that we are planning for toolkit. In the not too distant future, we are hoping to start looking at some of the core configuration and do a big pass on that to make it easier to understand and maintain. We think the type of approach you outline here is one of the cornerstones to making the config a bit easier to manage and we have also been talking about adding a framework to allow for advanced users to plug in their own "preprocessor" mechanisms as a way of connecting toolkit to other services and systems.

Thanks!
Manne

Comment actions Permalink
0

Rodrigo Rodriguez July 28, 2015 19:08

Hello Manne!

That's great! I'm glad to be helpful. We will be looking forward to this. Please, let us know if we can help, test or give you feedback anytime.

Regards!
Rodrigo

Comment actions Permalink

Please sign in to leave a comment.