In the next few articles I’m going to be discussing data management, including storage, backup and archiving. I’ll be looking at the different kinds of devices and the trade offs between cost, speed, capacity, and reliability. But first, we need to think about the different types of data that we need to wrangle. Different kinds of data have different storage, backup and archival requirements. So, the first data management strategy is to organize your data so that it is easier to use, share, and manage.

Most video post-production is completed on a project by project basis (movie, commercial), or is done by the episode. So it makes sense to start with a top-level directory for each project, with perhaps a sub-directory for each episode. This allows you to archive off older projects to make room for new ones. (I know this is pretty basic, but you would be surprised.)

Within each project directory, you should create a sub-directory structure that is roughly consistent from project to project. If the structure is being used by more than one person, document both the directory layout. This way, everyone on the team knows where to find and put things. Saves time, and time is money, friend.

Now that we have some basic organization in place, we can work on some refinements. Whether we are dealing with with online, backup or archive storage, its going to really help us down the line if we have a good system for separating out different kinds of files:

* Large media files that tend to not to change over time (videotape captures) vs. files that are smaller but change frequently (Final Cut Pro and After Effects project.) I keep all files for a given project together in one directory with this one exception. During production, I want to have media files on the fastest storage on my system. Also, I don’t want to be backing these files up as frequently (and with as many copies as) my project files.

* Things that can be rendered, compiled or recovered (say, from tape) obviously don’t require the same care as, say, the After Effects file for that killer comp. I keep all renders, compression outputs and intermediate files either in a separate directory, or I tag them using a color (more on this later.) One exception here is that the final master for your show needs to be kept safe.

* Files that contain sensitive information or that you want to keep private (like the spreadsheet that shows what you are paying everybody) should be separated into their own directory. That way, you can password protect the directory on a file server or on the web, or otherwise hide these files from the rest of the team. You can also encrypt these types of files before putting them in a common storage area, or backing them up.

* Your project directory structure should attempt to segment files according to function, so that each department can peel-off (replicate, check-out, sync, etc.) the portion of the project tree that is relevant to their work. So scriptwriters, video editors, demo writers, producers, audio techs, illustrators, flash programmers, etc. can work relatively self-contained in their own part of the tree. This may mean some duplication of intermediate work products, but its worth it. People have their own work styles and allowing for some flexibility within the deeper parts of the project structure is good karma.

Finally, you can use metadata tags in Windows Vista, or Finder Labels in Mac OS X to help automate file management tasks, and to communicate between members of the post-production team. Here’s the scheme that I use:

Red — This means “current” or “master”. Do not delete. When opening a old project directory, I can get to the most current version of the final output by looking for the red folder or file.

Orange — This means “original” or “source”. I use this to mark files that can not be easily replaced such as footage from all-digital video cameras. This is another “do not delete” flag.

Yellow — These are “intermediate” or “output” directories. I put renders and compression output here. If I badly need to reclaim space, I know that I can safely delete these directories.

Green — This means “complete”. Good for marking VFX shots, episodes, etc. that are finished so that the next guy in the pipeline knows that things are stable.

Blue — This is “work in process”. Essentially means: “back me up every chance you get.” It also lets others on the team know that someone is mucking about down there.

Purple — This means “private”. Sensitive material that needs special handling. I’ll exclude these directories when sending out a portion of the project tree to a contractor.

Grey — This means “archived” or “temporary”. This material can be deleted at any time to free up space.

While file and directory organization may not qualify as “hard core” IT, it is crucial to maintaining your sanity and seems to be a topic that many people find vexing. I’m very interested to hear what has worked for you.

Next: Storage.

Cheers,
Cameron

[i]Special thanks to my friend Sean Dillon, Architect, Content Management Solutions at Microsoft who pointed out the metadata tagging features in Windows vista. Also thanks to Sherman Dickman, currently at the Mozilla Foundation, for his insights.[/i]