0

Why are LocalStorage (and Step) entities invisible in Shotgun GUI ?

Sebastian Thiel

July 28, 2015 16:23

None

Follow

When a new project is created in shotgun in a studio that uses tank, it's just natural to have a tank setup created automatically in the moment the shotgun project is created.

This is best done with the shotgun events daemon, or something equivalent.

In order to setup the initial folder structure, ideally using tanks folder schema right away, one will have to know what the storage location shall be.

Shotgun already has the notation of 'Local Storage' or Filesystem Locations, which can be setup in 'Site Preferences -> File Management'.

When it comes to that, shotgun is amazingly consistent as such it will store this information in an entity of type 'LocalStorage', very similar to pipeline steps which are stored in a 'Step' entity.

Back with the new project, the vital piece of information I need attached to it during creation is the LocalStorage entity to use. That way, everything can be initialized in the right spot and will work like a charm.

In theory, this can be done by adding a custom mandatory entity field to the 'Project' entity, and specify to only connect 'LocalStorage' entities to it, practically though the entity type I am interested in doesn't show up.

In 'Site Prefernces -> Entities' I cannot even see these entities either to configure them.

A similar problem I recently had with 'Step' entities, which needed to have another field to specify the folder name tank should use when creating directories.

As a workaround, I will use a custom entity with just one field pointing to the LocalStorage entity I need. The latter will be setup via API to point to the right one, the former will be used as Entity to connect to the mandatory project field. This forces me to deal with redundant information, increasing my maintenance costs as these edits cannot be done by a user, and making the code accessing this information more complex than it needs to be.

Do you think there will be a way to make LocalStorage (and possibly Step) entities accessible in the GUI ?

Thanks

• Facebook
• Twitter
• LinkedIn

7 comments

• 0
  
  Benoit Leveau July 28, 2015 19:03

  Very quick note:

  > A similar problem I recently had with 'Step' entities, which needed to have another field to specify the folder name tank should use when creating directories.

  Isn't the short_name field of the Step entity already used for this?

• 0
  
  Sebastian Thiel July 28, 2015 19:03

  An amendment: I was able to edit the type of allowed entities to 'LocalStorage' in the project's new field through the API. However, the auto-completer in the shotgun web interface didn't list the existing LocalStorage entities for some reason.

  Using a custom entity type seems to be the only way around that seemingly artificial limitation.

• 0
  
  Sebastian Thiel July 28, 2015 19:03

  The 'short_name' field is the suggested standard. In case there is an existing shotgun setup, and a legacy folder structure to match, there is no one-on-one mapping between step short names and directories on disk.

  A new field is needed, containing an explicit name for the disk directories.

  Since the Step entity is hidden from the shotgun UI, the client will not be able to fill in the information in case he adds a new pipeline step, which will make tank folder creation fail at best. It's also not possible to specify multiple fields for tank to use as some sort of fallback in case one is unset.

• 0
  
  Permanently deleted user July 28, 2015 19:03

  Hello there Sebastian!

  Great questions as always! As you have already observed, Step and Local Storage are entities under the hood but "special" in the UI, making them harder to edit and meaning that stuff like the auto completion doesn't always work. At the moment, there is no good workaround for this other than the one you have already found; the API. I will pass your request to open these up to our product team – I know for sure that this is something that is discussed on a regular basis and we are most grateful for your feedback and use cases to help aid and inform the Shotgun product group.

  Having said that, on the toolkit side, I am curious to learn more about your project setup! The general idea with project setup is that a studio (e.g. a shotgun site) has a fixed number of local storages, matching their different file system key root locations. For a lot of our clients, there is only the primary local storage, which represents the one and only root point for their file system. But there can also be more complex multi root setups where you for example have one storage for main data, one for textures, one for editorial, review etc.

  Each toolkit project configuration template has a storage setup where it requires a certain number of storages. Our default starter config requires the primary storage but a more complex configuration may require several storages to be defined in Shotgun. During the project setup process, all the information you need to supply is 1) which config template to use and 2) the folder name on disk for the project root folders. The setup process will see which storages that the selected config utilises and concatenate these with the project name in order to get the paths on disk where things should be stored.

  You can control the setup project command via the api (see docs here) so if you have a shotgun event daemon running it should be fairly straight forward to call out to the tank setup_project command from an event trigger script. Is this roughly what you had in mind?

  With this setup, one way to set things up would be to add a custom enum field on the project entity and have enum values representing all your different toolkit project configurations used in the studio (VFX project, commercials project, etc) and as part of project creation, this is specified. In fact, I think there may already be a project type field that could be used for this purpose! Once the new project creation event triggers in the shotgun daemon, have some logic to map which toolkit template config to use for which type of project, and then run the tank setup_project API call.

  Would this be a viable setup you think? This way you don't really need to associate storages with each project. Apologies if I am suggesting crazy things here and quite curious to learn more about your use case! Let us know how we can help smooth out these wrinkles!

  Manne

   

• 0
  
  Sebastian Thiel July 28, 2015 19:03

  Hi Manne,

  Thanks for the elaborate reply !

  The suggested solution, associating a project type enumeration with a specific tank configuration that defines the used roots in yaml, would certainly work.

  However, the goal is to leave the client in a state allowing him to be self-maintaining. Any kind of configuration affecting the system must come solely from a GUI, which is the shotgun web front-end in this case.

  Therefore, entities and entity connections are viable, these can (usuall ;)) be handled natively via GUI. Anything that goes towards file editing has a red-flag. The latter isn't even problematic because yml files would be hard to use per se, but because you'd need to know so much more, like file location, exact syntax, and possibly about the inovlvement of git.

  The suggested solution boils down to maintaining multiple tank configurations, and managing their roots.yml files accordingly, which can't be done by the client.

  What can be done though is to have one, for a start, standard tank project configuration, and allow the client to specify a storage location. Multi-root is nothing anyone needs there (even though it's super-practical to have !), but what they need is a way to specify whether to use fast or slow storage, or crowded or empty storage. The company has various file servers with different properties, and depending on the project you want to choose the most suitable storage location for the entire project to live on.

  Having a project type linked with storage location via tank configuration is a mix of two orthogonal concepts, namely:

  * which server(s) to use for storage
  * which workflow and file-naming scheme to use

  Being able to mix and match is vital, mixing orthogonal concepts is **never** a good idea.

  Therefore I am taking as much as I can from the default 'setup_project' command, and modify some files after they have been created.
  All this code is open-source in the OSI sense, and can be viewed on github (https://github.com/Byron/btank).

  Best,
  Sebastian

• 0
  
  Permanently deleted user July 28, 2015 19:04

  Hello Sebastian!

  Thank you for your valuable insights, input and suggestions!

  As you point out, the storage decisions are currently baked into the config, meaning that if you want an alternative set of storages, you would basically need a new config to represent that. One of the reasons for the current design is to make it play nicely with Shotgun concept of Local Filesystem Storages.

  You'd be pleased to know that we are currently looking at some more long term measures, investigating how we can make the general configuration and maintenance of toolkit simpler and more flexible. One of the particular things we are looking at is how we can have more flexibility and simplicity when it comes to the storage concept - not just for file system locations, but going forward ensuring that toolkit supports cloud based storage locations and other environments where I/O resources aren't necessarily represented by a hierarchical disk like structure. This is a mid/long term initiative and nothing that will happen anytime soon, but at least I thought I'd let you know that we are very aware of some of the current complexity around the configuration and that we are actively working on trying to make it a lot easier to handle!

  Thanks!
  Manne

• 0
  
  Sebastian Thiel July 28, 2015 19:04

  I am looking forward to it !

  Good luck !