0

Local DB Cache/"Gateway"

Something which would inspire us to become more enslaved to shotgun would be if there was a way to keep a slightly out of date local instance on our network.

As a small studio we don't need nor can afford the stand alone option and like the web interface.  But some things are just a bit laggy for remote access.  A few things off the top of my head would be something like a camera server or other such scene management applications.   It would be nice to fold something like that into shotgun so that you have a 'camera' entity and then have the camera animation all baked into shotgun instead of a separate database running concurrently. 

In house we developed a pretty extensive scene management database for a specific type of project which can procedurally generate the final rendered scene.   But a scene will involve thousands of records (which don't actually change very rapidly but would be extremely slow over the internet.)

Similarly a Query for a few hundred entities would be a beast--it really needs to be hitting our local MySQL server to be usable.   If however we could keep a local copy--just for SDK hits that synced with the cloud every few minutes we could get the instantaneous responsiveness of a native solution but the flexibility and hands-off automation of the web solution.

I'm imagining a really simple WinService that's just running in the BG.   It would have a configuration for 'linking' the app to our web server.  A "refresh rate" for how often it syncs and maybe a few settings for whether web updates or local updates get priority. 

This cache wouldn't host the GUI.  So if you went onto the website and updated a project status it would update it online directly not through the cache.  

This local "Gateway" could also handle studio tasks and maybe have a python plugin architecture.   For instance if you wanted to link a frame sequence to a version the Gateway could automatically create a preview Qt.  It could handle the sorts of tasks that would be best handled 'client' side in prepping data for the web.   Anywhere you need processing or low latency this gateway could offer a beachhead service for Shotgun to call on. 

I'm just thinking this through as I write so going back to the Qt Preview.  Let's say someone links a frame sequence.  Let's say it's linked to a local frame sequence on the studio network.  In the next Sync cycle the Web server updates fields and settings on the local instance--but it also has a list of tasks:

"Hey Gateway, I also need a Qt Preview for a file on your network.   "\\path\to\frame\sequence_%04d.exr"   Then the Gateway would put it into its "task Queue" and start churning away on a Qt Preview.  On the next Sync the Task list would still have "Create Qt Preview" in its "InProgress" tasks.   "Hey Gateway, How is that Qt Preview coming along?"
"Done.  Here is a Qt Preview for Project-343_Sequence-010_Shot-0010_Version_10"

 

1 comment

  • 0
    Avatar
    John McLaughlin

    Hi Gavin,

    I'm pretty new to SG but have some thoughts about this anyway.  At Mass Animation we're looking seriously at Google App Engine to perform some of the tasks of your "gateway".   The best features of it are:

    1. Very straightforward Python implementation.
    2. Easy deployment.
    3. Highly reliable and almost zero administration costs.

    It's also likely that the latency of  ( GAE <--> SG )  is much less than ( "Client Workstation" <--> SG ) , so report generation should be faster -- especially if you can avoid SG calls in inner loops, and make "batch" calls instead.   Additionally GAE is designed with the assumption that you are implementing a massively multi-client web application and are doing AJAX style programming, so it possible to make interactivity seem less laggy.    One additional highly speculative future benefit is that since GAE will shortly be coming out with MySql support, it is possible that applications could take advantage of Google's own query caching or implement their own query caching.  

    I've done a few simple proof-of-concept apps, and I'm not seeing any red flags yet.   Of course for most productions there will be a host of security issues (both real and imagined) that need to be resolved, but I'm pretty sure I can solve those problems for our peculiar production model.

    I currently have two or three priority projects in front of this, but I'll definitely getting back to it in a few weeks.   I'll let the group know what's up as I move forward.

    John

Please sign in to leave a comment.