How I decided to finally buy shared storage

Data storage is a growing problem in 2022. You can buy big hard drives for very little money, drives that would’ve felt like science fiction 20 years ago. The average amount of space each person uses is growing (look at your phone storage), but so are resolutions and data rates. SSDs are getting faster and cheaper as well, but you’re still going to pay a boatload for options that are fast *and* large. Depending on what sort of business you have, the need for networked storage or external RAID will eventually creep up on you. That’s what happened to me this year. After 12 months of research, watching Youtube videos, and asking friends for advice, I finally pulled the trigger on a QNAP TVS-h1288x (Amazon link). Why? Let me explain.

HOW I GOT HERE

I have a general post-production, design, and advertising part of my business, but in recent years, digital platform delivery has grown to be at least a third of my yearly revenue. I work with half a dozen distributors who need a reliable lab to deliver new and catalog content to AVOD, SVOD, Amazon, Youtube, TV stations, etc., in the exact spec that’s required. They need it done properly the first time, and quickly. If I delivered a TV Show to Tubi a year ago, I might be asked to deliver it to Roku tomorrow. That means I need constant access to multiple large catalogs of content at a moment’s notice. ProRes files, H264 files, layered artwork, captions, metadata, trailers, all of it. Faster deliveries mean platforms, distributors, and filmmakers all start making money faster. That makes them happy, and I get paid, so I’m happy. What goes to one platform today could go to 5 more tomorrow. Slow deliveries also mean bad margins. Time is money, and I’m not getting any younger. In addition, I almost never receive files in the exact formats I eventually need. The amount of aggregation and communication needed to pull together all the proper files from just one film (artwork especially), alter or create what’s needed, and organize and name it properly for the future, is at least 1/2 the job or more, so having various long-term “buckets” of storage where all of one title’s content can be kept, changed or added to, backed up, and accessible *quickly* is important for me. I need to be able to seach a movie’s name and get moving fast. I don’t even need a team to have access to it (yet), but the problem remains the same, and as the scale of things grows, every bit of efficiency I have control over needs attention.

THE OLD SYSTEM

The old system comprised of various slow external drives (typically WD Elements or Seagate Expansion units) that I had clients buy along the way, and always two at a time, one to utilize on-going and another for a regular physical backup. All 8 external drives are plugged into 2 different USB strips (that have their own power) that then plug into my iMac Pro via two beefy USB-A cords. I use two 1200-watt UPS systems from Cyberpower (Amazon link) as a surge protector and battery backup. The hard drives (and now the NAS) have their own UPS, while my Mac and a few other accessories use a second UPS. Chronosync is what I use for weekly physical backups. I plug-in Client A’s 16TB BU1, the app checks for new files/changes, copies them, then I unplug and put it back on the shelf. Rinse/Repeat. This has worked for a few years, and I got up to 8 drives using this system with capacities between 10TB and 16TB (or in this case, 16 drives due to the backups), but last summer, I started having problems: My Mac was ejecting them due to power draw. It didn’t happen all the time, but if too many of the drives were too busy at the same time, it caused a panic. Not good, but at this unusual scale, is it that surprising?

In addition, if I needed to work with two titles that happened to live on the same drive, I couldn’t work on both at once. I have to do one task, then the other, and with days where multiple items are delivered by the end of the day, it means a lot more mental juggling and a lot more sitting and babysitting things. Compressing a ProRes file on a slow external to MP4, reading and writing to the same 100-175mb/s drive is bad enough, but trying two at the same time or opening a Premiere project file to check Audio Channel arrangements… not good. I find myself constantly moving some files around on my faster SSD storage (I own an 8TB OWC Thunderblade and have 4TB internal storage on my Mac) just to avoid some of the slow render speeds. The amount of time spent just waiting on files to copy in the last 12 months is too painful to admit, and the more back/forth you create in workflows that are unnecessary, the more chances you have for human error, forgetting where you left off, deleting items by mistake, etc. You can set up all the protocols and systems you want, but mistakes happen. I waited a long time to figure out a bigger solution because it wouldn’t be cheap to fix or simple, and the risk of getting things wrong would mean a lot of headache and expense.

However, I eventually realized that the storage situation wasn’t going to get better. Because most of the storage I need is being taken up by the delivery part of my business, these files can’t be deleted. I’m not like most friends in post-production, where the rotation of projects has them working with files and eventually offloading them to long-term storage or handing things back to the client. Tomorrow there will be more “forever” storage required for my clients and me. The amount of internet bandwidth I use monthly is extraordinary, with new titles arriving via Aspera/Dropbox/Google/WeTransfer or by mail all the time. Depending on the time of year, the entire floor of my office is eaten up by drives I have to return to different parts of the country. A networked storage solution is the only way to consolidate the power and access problems while giving me a much-needed read/write speed boost. I had considered direct storage solutions instead (basically the same sort of setup, but only attachable to one computer), but if I was going to spend this kind of money, I figured I might as well make sure I have the flexibility of a networked server. I can’t be sure that I won’t be a 2-person team in the near future or that I won’t need another computer around for distributed render processing that also needs access to the content at the same time. Better safe than sorry, and I hate wasting money. RAIDs always carry some risk, but if you’re smart, and buy the right things, have a good backup solution, etc., the investment can be worth it. So when 18TB IronWolf Pros (Amazon link) went sky high to $600+ in 2021, then down to $350 a few weeks ago, I finally jumped.

NOTE: I got the drives from Amazon, even though Newegg matched Amazon’s price, but this was a mistake. All 8 drives showed up in the individual boxes that the drives themselves came in from the manufacturer. They weren’t inside another box that was padded, they were just shipped as-is. I couldn’t believe it. One was broken and needed replacement.

THE NEW SYSTEM

Are you bored yet? I hope not, because I’m not done. I chose the QNAP TVS-h1288x for a few reasons. It had dual 10Gbit ports standard with four 1GBbit ports if needed. QNAP has an affordable Thunderbolt 3 card that doesn’t come standard but is easy to install in case I want it later (if not I have a 4x PCI lane to fill). It has 12-bays available on the front end for 4x 2.5inch drives and 8x 3.5inch drives, and while I only planned to fill the 8 bays initially, those 4 extra 2.5inch bays can come in handy later as I settle into the “NAS life” for either cheap individual SSD storage for caching (Davinci, Premiere/After Effects, etc) or creating another RAID storage pool inside the NAS if that ever interests me. The inside also has room for two NVMe SSDs, so 14 bays total. These are basically PCs that manage data, so reselling them is tricky as they age. If I was buying one, I wanted to make sure I could keep it for a long time. In conclusion, it was the best all-around option I could find for the sort of space I wanted in RAID6, with room to expand later.

Cloud backup was the last part of the process, and I didn’t even bother buying the NAS or hard drives until I knew how I might handle that. I wish there were more options for someone in my position, but Dropbox was it. All other companies out there didn’t provide the sort of space I needed for a cost per month that was affordable. Dropbox’s only option for “unlimited” space provides 3 users access (you can’t choose less than 3), and it’ll cost me $864 for the year. All other companies are on a cost per GB basis, from AWS to Backblaze to Wasabi. Their cost per month for the number of Terabytes would cost a few thousand per year, and the restoration process for some services is very cumbersome and expensive. By utilizing a program like Arq for the actual backup uploads and scheduling, I have everything I need to restore a massive amount of data in the event of a catastrophic situation, like a house fire, and NAS aside, that Dropbox account would allow me to backup my Home Drive daily, as well as my Thunderblade or other Externals as needed. QNAP has built-in apps that can do these backups for you, but I decided (at least for now) that I didn’t want my NAS connected directly to the internet. Arq can backup the NAS via my desktop’s connection to it. A NAS can be just as susceptible to attacks and malware as any other desktop. By watching great Youtube videos like this one, I took the necessary steps to protect my NAS as much as possible from the sort of problems that could take out this part of my business. Fingers crossed!

WHAT’S THE CATCH?

Well, this isn’t cheap. The NAS cost approx. $2800 (up from last year, unfortunately), and the eight 18TB IronWolf Pros cost $350/each (though they were nearly double that a year ago). Add tax and well… yea. In addition, I bought a 9th drive to have on the shelf, just in case I had to attempt an emergency RAID rebuild after a drive failure.

The other downside is that just like any other piece of gear, especially a computer-like system, it’ll depreciate fast. If it’s not going to help me make more $ or the same amount of $ faster or with more ease, it’s a money pit. In addition, even when buying the best NAS or the best enterprise hard drives on the planet, anything could fail at any time, for any reason, and even the best hard drives won’t last forever, so while I’m used to rotating out Mac systems every few years and handing them off to the used market, this NAS and these drives are probably mine until I toss them in the garbage. At some point in maybe 2 years, I’m going to wonder if it’s time to buy all new drives and start from scratch. Buying all new drives sounds crazy until you consider that once one drive goes from use or age, you start to wonder when the others will go too, and while RAIDs can be rebuilt, it’s a very long and intensive process that can easily cause other drives to fail during the rebuild. The business I’m in means I’ll have ongoing costs every few years at least to maintain things. Isn’t this fun guys? Who’s idea was this anyways?

THE BENEFITS

As soon as I started up Blackmagic Disk Speed Test and got between 800-950mb/s, I knew I had made a good choice. Yesterday I had Premiere exporting a feature-length ProRes file for a streaming platform, reading to and from the NAS, while Compressor created another, and Adobe Media Encoder worked on a third. All ProRes, all reading from and writing to the NAS. It didn’t break a sweat. In the past, that would’ve been a one-at-a-time process, or I would’ve had maybe two running, but they’d have to be titles that live on two different externals and reading/writing from the same slow disks instead of rendering to my SSD Desktop, would’ve been painful. Will it always save me time? No. I don’t always have that much running at once, but when I do, the savings could be an hour or two of fuss at least or simply mean less weighing on my mind as I juggle, like remembering to run Task A and immediately run Task B when that one completes. Now I don’t have to worry about what file is busy or where there is a bottleneck. This morning I was able to open up my Compressor instances to 4 and compress 4 episodes of a TV show at the same time, instead of one at a time. That was previously impossible unless it lived on my Mac’s SSD or the Thunderblade. Is that sort of thing always needed? No. Is it great when it is? Definitely.

In the world of benchmarks and render tests, I think we often forget all the little quirks of working in post-production that impact our day to day. As an example that’s relevant to this article, when you use some of these programs to compress or create files, it’ll often create a temp file, then combine it all at the tail end. When you’re dealing with slow drives, the render might not be that slow, but as soon as that file is “done” and it needs to create the actual final file right next to the temp file before deleting it, your read/write speeds are going to be a huge factor in moving forward. When you’re on SSD’s it’s pretty much immediate, but that’s not the case with slow external spinning drives.

Workflow is everything, and for my specific workflow and this part of my business, being able to have more things running at the same time is a real time saver. The amount of data I need access to regularly in order to pull all of this off isn’t something everyone has to worry about… but I do. What do I do once this NAS is full? Buy another one? Move onto a much bigger setup? I’m not 100% sure yet. Money is always a factor. 45Drives has some great systems available, but it’s all going to cost a bundle, and moving up to bigger systems means more power draw, more noise, and more fuss. I’ll cross that bridge when I come to it, and I’d need the revenue and cash flow to justify it anyways. For now, I literally bought myself some time and hopefully an easier ride in Q3 and Q4 of 2022.

CONCLUSION

Even if my specific issues or the scale of my issues isn’t relevant to you, I would consider looking at the various DAS and NAS solutions out there to see if they’d help your business. Maybe you absolutely need it, or maybe you’re just sick of the big stack of random HDs that keep piling up around you or the constant juggling of a million Samsung T5 or T7s in your office, and you just want to start consolidating things a bit more or cutting down on clutter. OWC has great DAS solutions like their Thunderbay series (their new Thunderbay Flex has CF and SD Card slots on the front!). QNAP has much smaller NAS systems than the one I purchased. Synology is also very popular. There’s enough competition for everyone to have options.

If you get nothing else out of this article, at least consider what your current backup system looks like if you have one at all. I’ve been horrified to find out that I’m the only one left with master files of someone’s film or slate of films. I’ve copied over the last remaining files of an old 35mm scan minutes before the HD died and fell into pieces on my floor. Yes, I know, scary. I work primarily in Film/TV, but it doesn’t matter what you’re doing. Figure out a system that works for you. Are you buying a critical HD? Buy 2. Pretend it’s 1 in your mind or when you budget, whatever helps. Don’t have a surge protector? Get one today. Can’t find a good cloud solution for personal data or family photos? Buy cheap externals, date a copy, and take it to your parents on Thanksgiving and stick it in a sock drawer. Mobile? Never carry your only backups with you alongside your laptop. Ever. Keep an eye out during Prime Day or during the holidays for Hard Drive and SSD price drops and stock up as a tax write-off. Protect yourself. There’s a reason we make it illegal to drive or own a registered vehicle without insurance… it’s stupid… so for the data that matters, get yourself some insurance. Building bad habits around data management will eventually hurt you or someone else.

Just like your hard drives, make some space in your mind for these things. A little bit of thought about these issues can go a long way.