Shotgun LMS (Local Media Server)

Now that we can playback from a local web-server the next step is a full local media server to cover all our needs :)

Currently Shotgun stores thumbnails, videos and attachments it the cloud. For many scenarios this is very convenient.
But for all those under strict security restrictions it means they have a bare shotgun without thumbnails, videos, attachments and without the awesome WebPlayer.
One approach would obviously be to host Shotgun locally and another to only have an internal media server while keeping Shotgun in the cloud.

It's obviously a bigger project but I thought the media server option could be explored here and if there is a lot of community interest Shotgun might consider implementing it.

So here is what I assume the current workflow to be:
- media gets into shotgun via the browser and via the API
- first it gets uploaded to the shotgun server and then forwarded to S3 for optional trans-coding and then storage
- for play-back/download the Shotgun back-end generates a temporary URL to allow the client to download/stream the data from S3 again
- if an attachment is deleted its removed from S3 storage

Here is what the workflow for a LMS (local media server) could be:
- the URL for our local storage is stored in the shotgun settings
- during browser or API based upload first an attachment ID is generated and then the attachment is uploaded to LMS via WebDAV/FTP with the attachment ID

- if trans-coding is required a request is added to a queue

- for playback/download the Shotgun back-end generates a permanent URL to the file in LMS and then it's downloaded/streamed from there

Regarding the LMS it would be best if it was a pre-configured virtual machine/docker image provided by Shotgun with components like web server, NFS/CIFS support, queue (like redis), trans-coding tools (like ffmpeg):

- This would prevent support issues arising from different LMS configurations
- Allow for trans-coding and image resizing to be done on the machine
- The queue would also allow to distribute the trans-coding workflow over multiple LMS VM's
- The machine would be stateless and mount an external storage for storing data and configuration so there would never be any upgrade issues (just the VM gets swapped)

Instead of using WebDAV/FTP the S3 protocol could be used which is already implemented by Shotgun for S3 upload but it would require the LMS VM's to be compatible with S3 through products like libre3 or riak-s2. Overall it seems less overhead to program the WebDAV/FTP calls then to depend on a S3 compatibility layer.

So would it be critical for your studio too? What other requirements does your studio have? etc.

Thanks for joining the discussion and helping LMS to become a reality some day soon!
Patrick

link to related newsgroup ticket

Nephi Sanchez September 08, 2017 02:57 Official comment

Thanks for the input Jesse!

Permanently deleted user August 27, 2015 20:26

I admit that the discussion on the implementation details of the LMS (Local Media Server) might be a bit lengthy.
But how about we as a community focus on the use case itself to see if its valid & useful to pitch a local media server?

What would be helpful to understand is the following:
Are there any other studio with a hosted Shotgun site who are not allowed to upload media containing client IP (thumbnails, images, videos, documents, scripts, set reference photos etc.) to the cloud? I'd assume there are quite a few.
At Pixomondo the policy is pretty strict which means we can't use the amazingly useful Shotgun Overlay Player and in most cases only have the smallest of thumbnails available.

Granted it's now finally possible to playback from a local web server but it doesn't yet solve the thumbnail & attachment issue and also involves extra programming and setup steps many won't have the time to do.

So the question I'm trying to ask is:
Would a LMS help solve some of your studio's challenges around cloud and IP?

Jesse Lucas September 01, 2017 19:32

This would be an ideal solution for us, as customers on a hosted Shotgun site doing work for clients with very strict security requirements.

Shotgun LMS (Local Media Server)

3 comments