0

Offline SG Connection and SGTK.

Robert Durnin

July 28, 2015 16:21

None

Follow

Hello again, I am looking at building in some redundancy to our pipeline tools and am realizing that much of the data being passed around is dependent on calls to our SG database. It sounds like an obvious enough position, but I am concerned that I will be unable to populate file io windows for asset management in the case that our SG connection goes down for any amount of time.

Are the file path parsing tools made available through SGTK dependent on SG to convert pipeline tokens into valid directory locations? I understand if a window which displays 'all active projects by name' might not be able to refresh, but I can't rely exclusively on internet connectivity for engine level file input/output.

Ideally, I could rely on templates.yml and it's tokenization schema to convert paths and find publish locations without having to derive context from SGTK or concern myself with updating entities.

r

 

• Facebook
• Twitter
• LinkedIn

6 comments

• 0
  
  Robert Durnin July 28, 2015 19:07

  I'm looking at this and thinking about how to pull the Template, PipelineConfig, and Context objects away from the library so that they could be instantiated without a connection to SG, or even rely on studio defaults. I understand that this puts at risk a certain amount of synchronization between entities and published files, but my priority is in keeping people working.

  r

   

• 0
  
  Permanently deleted user July 28, 2015 19:07

  Hello!

  Good question and a very healthy concern. You are spot on in your assumptions. The toolkit template system does not require a shotgun connection to run - this would simply be too slow. However, tools that extract data from shotgun generally wont work unless there is a connection (loaders, publishers, the file manager app). One way to put it is that the low level part of toolkit doesn't require a sg connection for its read operations where as most of the apps do. Folder creation will need a connection to shotgun, and therefore also app launching via our app launch app.

  Generally speaking, we have designed toolkit to work with partial connectivity, but not really for the case of longer outages, if that makes sense. We are currently starting to roll out a new generation of apps, most of which will be based on a caching framework, allowing apps to access shotgun data even when they are offline. This will hopefully help improve performance and reliability of toolkit for our clients that don't always have a solid connection, or the clients who are having a high latency or a low throughput. And apps will behave more like the core in that they wont strictly need a shotgun connection for their read operations.

  It's a tricky balance for us - Shotgun and toolkit goes hand in hand so at the end of the day we will need the rich nice metadata that shotgun stores so that we can build UIs that connect the data in sensible ways - yet at the same time we don't want the pipeline infrastructure to depend on an external resource.

  Does this answer your question? Please let me know if you want me to elaborate!

  Thanks! Manne

• 0
  
  Robert Durnin July 28, 2015 19:07

  Hey Manne, and thanks again for being so detailed in your reply. I believe I understand the breakdown of objects which do and do not depend on a Shotgun connection, although requiring a connection to the database in order to launch apps is a serious concern. We have a library of application launchers and low-level (fileio) classes which conform asset data and software to a local framework, and it is unlikely that we will adopt the SGTK 'engines and apps' paradigm. It is easy enough to assume that we are already 'in' an engine when it comes time to manipulate assets, and so our code base remains focused on publishing routines rather than extending connectivity to 3rd party apps. 

  While we are on the subject though I will share some of my thoughts:

  Storing tags to identify which scene versions have generated which publishes is as much of a link between a scene graph and the asset toolkit as we need. It doesn't make sense to program hooks to connect a third-party DAG/COM/DB to an asset database, and I've spent enough time trying to repair DAG/COM/DB chains to prefer a caching/stream manipulation workflow, like Katana or Nuke.

  The principal components we require to extend SG are the following:

  A python library for managing entities (something like shotgunORM).

  A python library for mapping entity tags to a file path schema (I believe we know this one).

  An event handler which uses the above libraries to keep the db current and enqueue update processes.

  Having spent some time working with SPI, I really cannot understate the ease of using their referencing system, and I am reworking shotgunORM to use a similar (human-readable) syntax. Extending reference tags to include the notion of 'context from location' could be done by parsing the single templates.yml and it's strings. It would then be easy enough to create PySide classes which hosted the reference tags, and custom data types for 3rd party apps that would allow for runtime conversion of the tags to paths. This last part is essential, and I will try and explain better with the following:

  The problem with bothering to manipulate the scene DAG, is that you ultimately have to store URLs on all of your scene input nodes at scene save. Every time you reopen a scene it requires a walking of the entire graph to determine which paths have been updated, and the process for updating is, at the very least, (annoyingly) interactive. Scene file parsing is slow, and the code base for checking inputs and data types swells the code base unnecessarily. In the case of a custom string type, which supports queries or expressions, tokens can be added for keeping versions current or to hard code them for processing. I don't suppose this removes the need for a connection to SG, but converting queries into file path strings can done without much network communication.

  I like that we are able to assign processes to drop down menus on SG entities, and application launching through the interface (in context) is also extremely valuable to us, but most of the data manipulation could be done with a decent event handler; and through a library for managing entities, the same publishing routines could be called interactively inside of host apps.

  I would like, ideally, to see a model more driven by data throughput (and able to guarantee access) that an engine-based one which requires exclusive connections (and doesn't seem to offer any additional data protection).

  Thanks,

  r
• 0
  
  Robert Durnin July 28, 2015 19:07

  Pardon me, I don't know what happened to my formatting.

  Let me know if I need to clarify.

  r

   

   

• 0
  
  Permanently deleted user July 28, 2015 19:07

  Hey there Robert!

  Thanks for your very long and interesting response! You touch on a number of really interesting topics and concepts that are typically found in high-end pipelines! It also sounds like you have an existing pipeline infrastructure in place in your facility already, so some of the things that toolkit is trying to help with may be of lesser value to you! Let me try to comment on the various things you bring up and hopefully my answers will help painting the picture of what potential benefits toolkit would bring to your setup:

  > We have a library of application launchers and low-level (fileio) classes which conform asset data and software to a local framework, and it is unlikely that we will adopt the SGTK 'engines and apps' paradigm. It is easy enough to assume that we are already 'in' an engine when it comes time to manipulate assets, and so our code base remains focused on publishing routines rather than extending connectivity to 3rd party apps. 

  Our apps and engines paradigm is merely a sort of implementation choice - our goal is to make it easy to build "pieces" in the form of apps and be able to reuse them across DCC applications (by providing a consistent interface through the use of an engine). Furthermore, we have extended this concept to add updating to make it easy for clients to pull in code updates and bug fixes!

  It is perfectly possible to run toolkit alongside another pipeline framework or library - in some cases studios have leveraged our Photoshop engine to provide a platform for them to integrate their own toolchain - in this case, toolkit merely provides a way to write python/qt code inside of photoshop - a platform that traditionally has been tricky to do custom development for, so our idea is to try to lower the barrier of entry by providing a toolkit engine that allows TDs and developers to work with technologies that they are familiar with! There is also a lot of API access generally, so you could start up an engine using the sgtk.platform.start_engine() API call and in that case there is no real need to actually use our app launchers if you don't want to.

  I guess in summary what I am trying to say is that there may be benefits in toolkit even if you have an existing pipeline in place. The benefits of the system is of course greater for a small studio who don't have any pipeline at all - and these studios are ultimately the type we are aiming at right now - however it is perfectly possible to run toolkit along side another pipeline and still use some of its concepts and technologies. We are trying to build a flexible base basically - a platform which allows both ourselves and our clients to evolve technology in a safe and open fashion!

  It sounds like you already have a pipeline design in mind - a strong, data driven, node based pipeline where there is deep application integration! Very exciting stuff. This is definitely a place where we ultimately will want to be with toolkit, but right at the moment our goals could perhaps be said to be somewhat more modest; we are trying to build small, configurable tools in the form of apps that makes it easy for artists to stay organized, avoid manual clicking around and making sharing and handing over data between departments and review easy and seamless. Toolkit could also be thought of as a way to seed the Shotgun database with rich metadata, allowing for the production to flow smoothly.

  I am sure our product manager Ryan, who is responsible for our roadmap and high level product direction, would love to chat about you about how you were seeing some of these node based concepts being integrated into Toolkit - I'll ask him to reach out if that's ok! In the meantime, it may be worth pointing out that if you feel the toolkit model isn't a great fit for you as it currently stands, it is also perfectly possible to integrate your pipeline with Shotgun by simply using the Shotgun python API. If you already have a pipeline set up in your studio, this may be a relatively quick process!

  Hope this helps!

  Thanks!
  Manne

• 0
  
  Robert Durnin July 28, 2015 19:07

  Hey Manne, and thanks again for taking such time in your reply. I would prefer, as much as possible to use the tools we are being supplied by Shotgun, and so making use of the existing code base is to my advantage. I was thinking to wrap the existing api classes to expose 'path to entity' functionality, and to create my own context class from it, but I will wait for a 2.0 release if it will save me from having to flush out all of the current dependencies on an active connection.

  Please detail any workflow options which will allow me to work with SG in an offline mode, as being able to work without an internet connection is a requirement for our facility. 

  r