Hi,
After reading the documentation I'm unclear as to how Tank would work with external artists, who don't have access to our internal network.
Tank offers version control, can it also be integrated into a version control system like SVN or Perforce, that would allow us to synchronize
files with external artists?
Thanks,
Ben
Has anyone successfully deployed the publishing tools yet? How would all this work?
It would be interesting to hear if anyone has already experimented with this type of workflow with Tank, but from our standpoint, we still have a bit of core work to do before we can reasonably cover the requirements. That said, we are making progress!
Tank v0.13 introduced the concept of a Pipeline Configuration, which allows a Project to have more than one configuration that can be isolated to specific users. While the main initial target use case is for developers doing testing outside of production, our thinking is that it will also be applicable to the outsourcing/remote artist use case as well. For example, you could have one or more Pipeline Configurations tailored to external artists, each with customized templates, environments, Maya/Nuke application launch paths, etc.:
These configurations would then only show up for the external artists in menus:
The Tank install, containing the core/engine/app/framework code can also be "localized" to a Pipeline Configuration, which makes them self-contained, and thus in theory easier to package up for an external artist as well.
As you can see, most of the work we've done in this use case's direction has to do with managing the configuration for an external artist. That said, our sense is that there is still a bit more to do on this component, namely:
Once we get thing squared away with external artists running Tank itself, the next step is to think about the data transfer/synchronization aspect. To that end, our hope is that the hooks we provide in Tank apps can be leveraged to do the heavy lifting. For example, in our default setup, when something is published there is a hook that copies the file to the publish location. For an external artist, this hook could be overridden to instead push something to Perforce or upload to an FTP site, say. Alternately, the publish location could just be a Dropbox or Box or other sync filesystem setup that would then get the files back to the studio.
So far, we're hearing that people are using a lot of different transfer approaches, so it would be great to pin down a favorite (or maybe a couple favorites, at least) to focus on initially. With any approach, there may also need to be a bit of bookkeeping work to make sure that paths created by an external artist map back to the studio properly for when an internal artist wants to use something generated by an external artist, and vice versa.
In any case, we're really excited about this workflow, and we think there is a lot of potential for Tank to help make this easier for clients. We'd love to start getting into more specifics with clients who are interested in this workflow so we can start fleshing out our plans and sharing them with the group!
Ryan.
That's awesome! I can't wait to dig into this. But one key question -- what files\plugins\tools do I have to send the external vendors to set up Tank? Do they run the same install as I did when I was setting up Tank for the first time, or something else?
Thanks!
As far as the files that would actually be required to run Tank (i.e. independent of where publishes are sourced from and are written to) are concerned, my initial thought would be to have the process be more like cloning a Pipeline Configuration within a Project and localizing it. That way, all of the installed components (Tank core, engines, apps, frameworks) and configuration could be "bundled" together and external artists wouldn't necessarily need to manage both their own Tank install as well as a Project-specific configuration. Also, an external artist might be working with more than one studio, each with a different Tank setup. With that in mind, it would probably be a lot simpler to just have each Project operate as a "localized" setup that doesn't have to account for what any other Project is doing.
This is starting to get into future development, but ultimately, we would want/need to have a way to "bundle" up a cloned Pipeline Configuration and link it to a given external artist in Shotgun. Maybe that just uploads the bundle to Shotgun and lets the external artist download it from there? It feels like a bunch of options would be possible. Then, the artist would download the "bundled" Pipeline Configuration and install it to a place of their choosing (or maybe they don't get to choose this?), which would also update Shotgun.
In any case, it feels to me like it will be beneficial to allow for the Projects to have "localized" setups and not have to force an external artist to manage their install the way a TD or admin would want to within a studio, or at the very least make it so that stuff is optional. How do you feel about this?
Ryan.
Long term, it would be fantastic to have a bundled configuration that can be installed by even the most non-tech-saavy remote artist. This configuration would, in my dream scenario, do the following:
I have no idea how feasible all of the above is - I'm not a developer - but I'm pretty sure that if you could somehow crack it, a lot of facilities would start to look very seriously at the possibility of employing a geographically-distributed workforce for at least some of their work. Which would be a bit of a game-changer, especially if it was nicely integrated with cloud rendering solutions like Zync...
I would be just interested if artists who are in the studio can download/check out a version of a maya file to an external drive. Go home work on it. And then when they return to the studio they have the ability to save it back into the system as a new version. Is this possible with the default configuration? I guess I am asking how would someone add a newer version of the files to the server through Shotgun and file manager or some other method outside of just copying it into the mnt/projects/.../work/folder.
Hi Stephen!
For your go-home-and-work-from-an-external-drive use case, would this be the only way artists worked? If so, that is potentially doable now, where you use a multiroot configuration to point the work area templates to the external drive, and the publish templates to your primary filesystem. When artists would do work, it would all go to the external drive, and they wouldn't be able to publish until they returned to the office and could see the primary filesystem. Even if they might do work at both places, that could potentially be handled by having multiple pipeline configurations, one that puts work areas on the primary filesystem and another that puts them on the external drive.
What I'm describing is more on the "possible" side but would have some rough edges (stuff we hope to address soon!). For example, an artist could try to publish from home, but would get an error because the primary filesystem wouldn't be accessible. If there is a separate pipeline configuration, an artist working at home might still get an ignorable error about the primary pipeline configuration not being accessible.
That said, it could be an interesting experiment. Let me know if you want to give this a shot and I can whip up a sample config for you that tries to work this way. Your use case is a little bit easier in that the artists will at least initially have access to the primary storage. But hopefully we will be able to solve that soon as well...!
Ryan.
Hi Guys,
We're also very insterrested in how we could share work outside the studio and get it back into tank with screening rooms QT's etc.
We outsource work all the time - but mostly to small studios or induviduals who has there own pipleine. So instead of getting them a full Tank / SG intergration, we are looking for a good way to get work in and out our pipeline. I was thinking a little more simple.
As a begining connecting SG with our FTP. So when I assign tasks to a external artist / studio - it will make the plates availible to them via Symlinking the folders where Tanks put the plates/footage to a FTP login.
But what about getting renders back in ? Would it be easy to do a tank App - that checks a directory for files (framestacks) and add each sequence it finds to a shot > version in SG (based on the folder naming convention) ?
I also considered letting external artist / studio's add versions via the RV submit tool - but then the Path to frames has to be changed somehow by a script after the files has been transfered to our ftp server.
best
jonas
Great to see so much interest in this - it really helps validate our thinking as well! Jonas and James, I'm going to consolidate my reply to both of you, hope that's ok...
Jonas:
Much of what you're describing does feel doable with some additional apps. Perhaps something like:
Admittedly this is a bit handwavy so I'm mainly trying to get a few more details out there so we can figure out the right scope!
James:
Yes, we would definitely have to account for the use case where a person works both in the office and out, and we would need mechanisms (like your "working remote" flag idea) to help dictate what should and shouldn't be available based on what someone is trying to do. This is the kind of polish we will aim to do as we also look to ironing out all of the technical details as well.
You also touch on a key area of how we manage what Tasks happen where (i.e. in the office or at home), as well as what data is where. I've been working on a proposal for some Shotgun schema additions to help track this, and I'm hoping to have it presentation-ready in the next couple weeks. The idea here is to lay the groundwork for people to track if a file is at a given location as well as which locations it exists. Then it will be down to the file sync to do its thing and update Shotgun accordingly. It would be great to get your eyes on the schema design and our hope is that it will be another big step in helping us make this workflow easier to manage.
Ryan.
Wouldn't external-Ryan always have that pipeline configuration though? What about a person working inhouse that can work remotely at anytime too?
A flag in the user dashboard to set 'working remote' ?
Definitely being able to pull down a bundled config / project tank setup would be best, directly from shotgun would be great
As for the file syncing...
It's either user managed, or ;
Correct me if I'm wrong, but you'd need to track the submits / publishes to the external primary locations back to the `local' inhouse primary locations. Even if the user was flagged as remote at the time of publish, the publishes would be still updating the db so the shotgun pages would update, but nothing in the inhouse file system would, so you'd have files being reported as existing but failing to load on application launches in house.
If there was a way to mark those as remote submissions, awaiting files in house, say an `internal ' 'external' column in the files areas, it'd then be down to the file sync itself?
As suggested some kind of custom transfer app that connects into the internal file system (both have the same paths, but the internal has placeholders), that then transfers the user selected files back up to the main file server (have fun with that one).If you could boast a file sync system of some kind that doesn't rely on an external sync tool like dropbox, it'd just be another notch in the belt really..
Hi Ryan,
Thanks for your reply ... I like the idea about right-click on a plate publish and say FTP to ___ Vendor - We have our vendors as a disabled person entity, so I guess it would be easy to to have a field here, with the path to there directory on the FTP. For getting stuff back in, you're right about the watchfolder could be tricky with framestacks - maybe it would require the vendor to submit it through a Vendor login or a modified RV submit with upload functionality or something like that. But I also understand that a more manually workflow could work where we, on our side, would just take the uploads into RV and submit them from there.
We really wanna jump start this workflow, so I guess it would be good to know if you're planing for some overall vendor / external artist apps in the nearest future. I guesss we could easily do the FTP to ____ Vendor ourself - one question in this regard : would a Tank app be the best way to do this - and how would we be able to ask which vendor to send to from within the SG webinterface ? Our should we do this via eventstream and have a field in the publishes with a link to the Vendor (people entity)
cheers
jonas
Hey ya
Would be great to look over the scehma.
Def great to see this being looked into :)
Hi
Very interesting topic. I'm trying to set up and develop this exact workflow, our team consists exclusively of external artists. I've managed to hook in to the publisher app to seamlessly download the required file from our ftp and then when the artist is done, upload it upon publish. Also they can render from maya/motionbuilder and upload a version to shotgun for feedback.
I'm a bit stuck though on a security issue. Since we need to give every artist the shotgun.yml file, which includes the api-key in order to communicate with shotgun, we give every artists all the permissions of the 'API Admin' permission role. Should the key get in the wrong hands that person will gain full access to all our shotgun data. Is there a way to restrict a user of the shotgun api to have the same permission rights as its shotgun user? Or at least only access to one project? Is there a way to create new permission roles which can only access one project and then use that role for a project exclusive script/api-key?
Cheers,
Nils
Hi Nils.
Great point! This is something we've been thinking about as well. On the Shotgun API side, the v3.0.15 introduces the ability to log in as a HumanUser via the API so you don't necessarily have to use an admin API key:
https://github.com/shotgunsoftware/python-api#changelog
This is something we're looking to leverage for Toolkit as a whole, where the user could "log in" to Toolkit + Shotgun and have the setup as a whole take on the permissions for that given user instead of the catch-all API key. We're thinking that Shotgun Desktop (previewed in our last webinar) could be our vehicle into this, what are your thoughts on this approach? Any potential gotchas with making it so the security options for Toolkit align with the permissions for a given user in Shotgun?
Ryan.
Hi Ryan!
That's excellent news! Shotgun desktop looks amazing, something I've really been missing. I think it's absolutely the right approach to provide a login and launch interface like Shotgun desktop and I don't see any potential problems with having the permissions align with a given shotgun user, other than taken it into consideration when writing the apps.
Does Shotgun Desktop also support updating the configuration and apps? Like checking for updates to a projects git repo and pull the changes?
/Nils
HI Nils.
We definitely plan on Shotgun Desktop having the ability to update both the project configuration and the installed components (i.e. the Toolkit core, engines, apps, frameworks and of course the Desktop app itself). But, it's not in the pilot build yet. Our sense is that the feature to update the installed components will come first since we already have the "app store" mechanism to check for and distribute new releases of stuff. With the configurations, we have to do a bit of homework on how those will be shared that we're hoping to share on the forums and/or in a webinar soon. A Git repo, as you describe, has been discussed before so it's encouraging to hear you say that...! :)
Ryan.
Hello Ryan,
We're working with multiple studio on a full 3d feature film. I try to find the best way to share my master configuration to other studios.
Currently, I've pushed all config files to a github repo and I'm wondering what's the next move. I tried to clone pipeline configuration but It's based on my local files. Then I cloned the git repo on a external local server (to testing) but there's no way to add a pipeline configuration and browse to the right local directory. Finally, I launch tank setup project and take git repo as [tk default_configuration] but I can't assign a new pipeline configuration to this new local copy. Do you have the right way to solve that issue?
Thanks
Denis
At present Denis the way it appears is you have to make sure you replicate the setup pretty much exactly at each point.
Essentially you're setting up a studio at each location.
To that end your primary volumes should match, as should any mappings etc for your software/studio and software/config folders
I've used github to sync the base configs but when I tried to add studio it was too big to sync and I really need to get more time in the day to rectify that aspect as it helps with base application updates etc.
Anyway to that end what we have going on here is:
Primary: I:\
Secondary: K:\
our dev/tank drive is T:\software
So the studio and the configs reside in T just to keep them cleanly away from prod servers etc
From memory I just downloaded and setup the first tank, then I githubed the configs to the repo (not studio atm) and made sure those git repos were directly in the T:\software
Then from there I zipped up the studio folder and transferred that over to the new location
Setup I: K: T: drives over there, put the updated studio into T:\software, then pull the repos into the T:\software correctly so the latest configs can be pulled directly to the servers on release
we are working in a fairly basic setup so each dev artist works on a local T:\software setup (which unfortunately means they have to copy their studio folder from the server at this time to a local drive that is mapped at T on their dev boxes) but this means they can grab the repos from git hub and work locally and push to T:\software\configName_sandbox for testing. When we're all happy that the latest configName_sandbox is working okay and everyone has commited to the gihub repo, we can
a) Pull to the server production sandbox for testing or
b) copy the sandbox over to the release config, clean up any pathing requirements etc and then release sandbox to the primary configuration and commit to github and pull down to the main servers T:\software\primaryconfig
All locations are using the same pipeline configurations and script keys from the base project as well so it's pretty clean there too.
This way anytime we need to roll out a release, as long as the base setup is done, once we commit a release to the main T:\software\primaryconfig all we have to do then is pull the repo to the main T:\ server drives at each location to remain in sync.
The drama comes when you have to update tank core. And it runs through all the apps and alters all ya hooks etc :P But it's manageable.
Anyways I figured I'd do a little dump in here just in case it helps ya :P
Hi Denis and James!
Very interesting stuff here! I have a few follow up questions for both of you...
Denis, are you sharing the same Shotgun site with the studios that you're partnering with on the feature film or does studio (or at least some of the studios) have their own Shotgun site(s)? For the publish data itself, are you structuring things like James is, where all the different partners can see the same primary storage so their publishes go right to a consistent spot everyone can access them from?
James, do you think it's a bad thing that the studio directory ends up being localized for everyone? By that I mean, what if it was just easier to do that in a built-in way?
One of the things we're excited about with Shotgun Desktop is the new ability it grants us to manage where configurations and installs live, potentially making it easier to localize them in a consistent way and place (ex. /Applications/Shotgun on Mac, C:\Program Files\Shotgun on Windows, etc.). While this won't all be in the initial release of Shotgun Desktop, we're starting to think through the planning and design for this feature set and we touched on it a bit in the last webinar (around the 28:10 mark):
https://support.shotgunsoftware.com/entries/95443087
The idea here is that we can expand on the current install options to also support all-local setups as well as hybrid setups where some users are localized and others (maybe those who actually work in the office) use a shared setup. The attached framegrab is the most relevant slide from that part of the webinar, showing how each user would have Shotgun Desktop, but some would point to the shared install and others would have their own install, with each sharing the same main data set (although this could also potentially be move-able as well).
The current idea we have to try and better support what it sounds like James is doing would be to help share configurations, possibly through Github (as Denis is doing) or through Shotgun itself (or perhaps both can be options?). Shotgun Desktop could help facilitate it, automatically grabbing any updates as they happen. But, we'd only share the configuration, and any of the installed components (core/engines/apps/frameworks) would then be pulled down separately and dynamically as needed. This way, we wouldn't need to always store a big mass of data (aka the studio folder), which can be a bear.
Anyhow, hearing about what you guys are trying now and how you'd like things to work is great for us. It really helps refine our design work, hopefully we'll have more stuff to share for feedback (or even testing) soon.
Ryan.
>>James, do you think it's a bad thing that the studio directory ends up being localized for everyone? By that I mean, what if it was just easier to do that in a >>built-in way?
Yeah it can be a bad thing. Because the configs rely on applications from the app store, and updating those at each hub then runs through the configs at each hub and alters all the shot_step.ymls etc as necessary, the issue I found with this is that each location then commits those changes to github and it can potentially create issues with the repos.
Studio I'm not githubbing atm due to the size limit and I need to look into this further when I get some time. But I think this would semi avoid that issue if I could push the studio apps through github and then the configs from one location.
So essentially I do a core update / application update remote, then transfer any new applicatiosn manually to the other locations, do a local tank core update *no apps* then commit and push the one config from my location out to everyone else and then check to make sure they're all working okay with the latest applications, then do the release from there after problem solves etc.
>>One of the things we're excited about with Shotgun Desktop is the new ability it grants us <SNIP>(core/engines/apps/frameworks) would then be pulled >>down separately and dynamically as needed. This way, we wouldn't need to always store a big mass of data (aka the studio folder), which can be a >>bear.
Okay I've attached a couple of images as a means of a brain storm / discussion. The Current.png is the way I kinda see it atm, the Anticipated.png is my brain dump just ignoring any mitigating factors for now..
I can def spend more time cleaning these up and making them clearer so apologies if they're a little obscure or not as thorough as poss, under the pump and all that atm.. :)
cheers
Ryan, will Shotgun Desktop be available on github in any sort of unofficial and unsupported way before you release it more officially? The things I'm developing at the moment would be very dependent on how shotgun desktop is working and it seems your test pilot pool is full :(
Cheers
I'm looking at the same kinda scenario, but maybe in a slightly simpler form:
I'm thinking more in terms of remote artists and not so much sharing work/data between studios. I would imagine an artist to VPN into the studio and thus having access to all the data (and having all the behaviour as a internal workstation). In order to actually make it workable over the web I was imagining a 'localize' option for assets which would download the selected assets to the local disk.
I could see the same functionality being useful within the studio environment (and have seen it at some studio pipelines) so you're less affected by network slowdowns...
Has there any progress being made on the endeavours mentioned in your previous posts?
-Bernie
Hi guys!
Just one question about the project root with the localized configuration - shall it be in the same place as defined in the website configuration or there's a way to customize it's location for an outsource work?
Andriy
We're looking at the same setup as Bernie. External user that has full access to our internal servers, but for speed improvements would like to work on files locally and the publish them to the main fileserver.
-atle
Anything new on this?
