So I run a video production company. We have 300TB of archived projects (and growing daily).
Many years ago, our old solution for archiving was simply to dump old projects off onto an external drive, duplicate that, and have one drive at the office, one offsite elsewhere. This was ok, but not ideal. Relatively expensive per TB, and just a shit ton of physical drives.
A few years ago, we had an unlimited Google Drive and 1000/1000 fibre internet. So we moved to a system where we would drop a project onto an external drive, keep that offsite, and have a duplicate of it uploaded to Google Drive. This worked ok until we reached a hidden file number limit on Google Drive. Then they removed the unlimited sizing of Google Drive accounts completely. So that was a dead end.
So then we moved that system to Dropbox a couple of years ago, as they were offering an unlimited account. This was the perfect situation. Dropbox was feature rich, fast, integrated beautifully into finder/explorer and just a great solution all round. It meant it was easy to give clients access to old data directly if they needed, etc. Anyway, as you all know, that gravy train has come to an end recently, and we now have 12 months grace with out storage on there before we have to have this sorted back to another sytem.
Our options seem to be:
- Go back to our old system of duplicated external drives, with one living offsite. We’d need ~$7500AUD worth of new drives to duplicate what we currently have.
- Buy a couple of LTO-9 tape drives (2 offices in different cities) and keep one copy on an external drive and one copy on a tape archive. This would be ~$20000AUD of hardware upfront + media costs of ~$2000AUD (assuming we’d get maybe 30TB per tape on the 18TB raw LTO 9 tapes). So more expensive upfront but would maybe pay off eventually?
- Build a linustechtips style beast of a NAS. Raw drive cost would be similar to the external drives, but would have the advantage of being accessible remotely. Would then need to spend $5000-10000AUD on the actual hardware on top of the drives. Also have the problem of ever growing storage needs. This solution we could potentially not duplicate the data to external drives though and live with RAID as only form of redundancy…
- Another clour storage service? Anything fast and decent enough that comes at a reasonable cost?
Any advice here would be appreciated!
NAS.
Over the last 24 months I’ve built 300TB (a mix of 10 and 14TB disks) for $2500 in disks. I could do that right now for $2100. A 18TB LTO9 tape is more expensive than what I’m paying per TB for 14TB disks.
$700 in hardware to build the NAS with 25 bays.
Glacier would cost you $1080/mo in storage fees alone (300,000GB @ $0.0036) not including the $0.09/GB to get any data back out. Deep Glacier is less (by half, for storage), but comes with strings attached.
Don’t forget to factor in labor hours of what it’s going to cost you to maintain a tape library or a local server in general.
Are you charging clients for long term storage after a project is complete? If not, you should be.
AWS Glacier Deep Freeze is designed for this. Something you access a couple of times per year if that, and it’s $.99/TB/mo. Price that out compared to a $10k NAS or tape backup that will still need consumables like drives and tapes, and it might be your best option. There are costs on retrieval, but since as you’ve said this is archive footage that customers might request you could pass that cost down to them.
Tip: AWS Snowcone & AWS Snowball are less expensive for data-out when you need to move many TiB. There is no time-limit on how long they can be rented.
Have you considered Amazon S3? It’s made for enterprises with unlimited storage, a lot of pricing options and could save you a lot of headaches long term.
Backblaze B2 is cheaper by a long way.
s3 is designed with high availability and high throughput in mind, op needs a cold storage solution like aws glacier or azure cold storage. but even that is not cheap
Yeah we looked into it. But as subven1 pointed out, it’s a brutal monthly cost.
You have 3 issues, online archive of past projects, long term (offline) storage & client access.
LTO is your long term solution for offline archive of projects. Depending on the average / largest project you might want to do 1 project per tape so LTO7/8 sizes. Scales really well, multiple copies, etc.
For the online storage, a NAS is really the only option. How it’s sized & configured comes into play. You can go cheaper with used enterprise gear, but then you’re dealing with more disks & higher power bills. Fewer larger disks can help with the power bill & noise levels.
Splitting things between a read-only share (of things that have been archived to tape), and a normal working share would help on the workflow.
The catch is what you do for client data exchanges. Giving them access via Dropbox is nice, but you need better housekeeping around data. Once the 1 year grace is over, what’s the size they have committed to? While self-hosting a client accessible share is possible, there’s ongoing costs & I would be cautious around exposing the NAS to the internet directly.
I’ve worked for several production companies that have similar or larger archives (one was well into the Petabyte range). LTO is the way to go. It is the cheapest option for very large archives, and if the tapes are properly stored, they last a lot longer than hard drives sitting on a shelf.
The real way to do it is a tiered archive, where everything goes to LTO, you have more recent media (1-2 years old, depending on project length) on hard drives, and current media (still in use + past year or so) on a NAS. LTO is still your primary archive; everything else is for easy access to media you’re more likely to need now or in the near future.
I don’t know that I’d take on tape with your use case. There’s a good bit of tech debt involved there.
NAS (either bought or built) + Amazon glacier or Backblaze for cloud archival backup.
The NAS (including drives) will probably cost you $7000-8000 USD for 400ish TB of storage with room to grow
It was easy to give clients access to old data directly if they needed, etc.
I hope you charge for this. It would help to offset your storage costs.
What drives and NAS would give you 400tb for 7-8k?
Yes I too am interested and would purchase this solution
300 TB in Backblaze B2 using their online calculator is $21,600 USD a year. I’m sure you can build / expand a new NAS every year for the similar prices. But then you have to deal with the overhead of managing it and replacing disks.
Wasabi has their Reserved Capacity Storage where you can get discounts if you commit to a minimum amount of storage. According to their site the absolute minimum to qualify is 25TB.
I suspect 300+ TB will get a decent deal.
How often are you actually needing to access the 300TB? If 250TB are “cold storage”, then LTO is the way (you can rent the readers usually, rather than buy)
If you’re needing to have access but not edit from, NAS is the way, 300TB wouldn’t even be THAT expensive (still expensive), just slow to move to, but once you’re up and running a decent rig should last years.
If you’re needing to access all 300TB, then you’re looking at a LTT style NAS that needs to handle read and write from multiple users at a time, and that’s gonna be the real $$$.
I feel like you might do well from a mixture of all of these. A smallish NAS for day to day/project use, and once that project is done you move it to the big “slow” server for onsite backup, and once every 2-3mo you rent the LTO drive and load up a few tapes, and ship them off to the void for offsite backup and cold storage.
idk you can rent them.
oddly thank you for that info.
Also idk where OP is finding LTO 9 for $20k, HP makes a LTO 9 drive for $4500
Just copying from a response above:
This is only for archived projects. But we’d probably still need to access ~10-20TB of that data relatively regualry to update branding, or change edits, etc. Saying that, as mentioned in the OP, if we went tape or cloud, we’ll likely have a physical local copy on an external hard drive for quicker access. We just need a redunant back up of these archives.
If we went NAS, I feel like maybe we could get away without the redundancy? Risky…
Which 20TB? The last projects or random?
More random than I’d want. Generally predictable though.
If we went NAS, I feel like maybe we could get away without the redundancy? Risky…
That’s the thing, you could, but it wouldn’t be best practice. At the end of the day the 3-2-1 rule applies to any data.
I know it’s a hard pill to swallow, but ideally you’d need both a NAS (I’d go with Proxmox on a PC) and the tape backup for that NAS to ensure the safety of the data.
However. Backblaze may take the spot of the tapes - unsure if the NAS as well. Have a look at their offer and see what fits your budget. I would personally go with the NAS on site and backup it daily to Backblaze. Note that Backblaze B2 says something like 6$/TB/month which amounts to about 21600$/year which stings but then again it’s safe and it’s the best value (all the competition seems to be more expensive).
I mean; the NAS would have some built in redundancy via RAID 5 or 6 or whatever, but you wouldn’t have an offsite backup. What you’d wanna look into is something like Backblaze B2, but even that is going to be $1800 a month, so at that point I would say build a 2nd NAS and pay for it to be in a data center, that would only be a couple hundred a year, or even just run it at your house and run a nightly backup.
Listen to me. Here is the pro solution. Get yourself something like Fujitsu Eternus cs800 plus Fujitsu lto tape library. Contact sales team and tell them how many data are you going to put there. Result will be all the data available quick if they reside on disk cache, or little bit later if need to be pulled from tape. From your point of view data will be available from mounted network share and transparent in terms of technical magic behind it. Basically - imagine yourself an infinite folder where algorithm is moving data to and from tapes, keeps them healthy, refresh and consolidate when needed. 20 tapes each 12tb plus dedup is like 0.5 PB of data. And you can always duplicate tapes and move to external location. Even if somebody would stole everything from 1st location including hardware, you can get data back.
https://youtu.be/lO-SAzFaN18?si=Rp0mvidHMxBFNedC
https://youtu.be/JHVSoJDZ06U?si=_7kmEUZNKc3UDfRK
I’m by no means qualified to give you an actual answer, but these videos may be of some use to you.
Yeah sweet. I haven’t checked in on the Slow Mo Guys storage setups in a while. I’ll have a watch.
Really depends on how often you need to touch your data. Tape has high upfront cost (4-5k $ for a LTO-9 tape drive + ~3,5 $/TB in tapes) but you don’t have to worry about archive space anymore. Otherwise, NAS space (if you selfhost) is ~15 $/TB + a server which would also be slightly above 5k right now to store your 300 TB.
You also have to calculate the power cost of a NAS.
From what it sounds you want a NAS and Tape Archive.
So get a device which holds your working Projects, you mentioned arount 20-40TB which is no problem nowdays. Can be done for under 1k with of the shelf stuff.
And Tape backup for stuff you dont need regularly. Maybe chose an older generation of LTO I would look for something that can hold about 1 Project per Tape or the likes of it. LTO5 is pretty cheap used, ca be had for 500 Bucks but is only 1.5TB per tape.
Disclaimer, with LTO never look at the compressed NR, its for compressable data only which video is not. Thus with LTO9 you will only get 18TB
This.^
2 small NASs + 2 LTOs (LTO5 may be sufficient for your individual projects, but you also need to backup the NAS, so at least LTO 7 or 8)
Yeah we’ve got a solid situation for our live projects. Each of us work off 40TB thunderbolt raids with local external drives as our backup and live online backup to Dropbox.
This is for our archived work, but yeah of that, we access around 20-40TB fairly regualrly. Good to know that tape won’t compress video data at all!
NAS is sounding more and more like our best bet.
For what it is worth in the NAS:
20x 6TB HDD - $1,400 ($11.66 per TB)
10 x18TB HDD - $3,500 ($19.44 per TB)Hopefully deals like these can keep the cost of your NAS down a bit
Not to be rude or anything, but External RAIDs individual to the user is not really a solid soulution. It may work for 1-2 People working on one project at a time. But it just does not scale. What if someone needs to acces files of that project? they move the raid or plug their laptop on a differen workspace? Not really a great soulution IMO.
Like you say in the last part having a NAS with maybe a bit of room to grow sso 100TB might be the best option that way everyone can access the data and work accross projects. And more importantly it would offer work from a different place in the office or even work from home.
Yea with tape the compressed nr are very missleading. Thats a best case scenario where the files compress 2:1 with TAR+gzip which it literallly never does. Bestcase I have seen was 1.2:1 on a folder consisting of config files. Basically nothing nowdays is compressable you will interact with, except textfiles depending on format. So its best to always asume the raw space as the space you get
Haha we’ve been this way for 12 years. Certainly not ideal if we scale. But we won’t ever. 4 of us ever needing access. And transferring over the network is not an issue. NAS is too slow for most real time editing. 10gbe is fine but still fairly slow. Those raids will soon be upgraded to SSD raids for each editor. Thanks tho…
IMO it depends on how organized you are and how often you need to access archived video.
LTO-9 is cheaper per TB (haven’t run the numbers, but on the order of 100s of TB it’s almost definitely true) but relies on someone physically finding the right tape and putting it into the system (unless you shell out for a very expensive automated system). Not good for fast access, but cheaper for expanding.
If you need fast, automated access I’d recommend the NAS option, but keep in mind that it would be in one physical location. A fire or flood and you’re fucked.
Plus, since the cost per TB of tape is so much cheaper than HDD, expanding your archive is probably much cheaper with tape (keeping in mind the organization/automation aspect)
Did I read somewhere there are two work/office locations?
You need ot think about how often you need to access the data. If it’s once or twice a year, then the added overhead of having to find and load a tape wouldn’t add up that quickly and IMO should be acceptable.
However, for projects you currently work on, you’d want hard drives and/or SSDs, preferably on a network, I suppose. Unless all your in0flight footage resides on the computers you edit them on (in which case I hope they have redundant storage).
Also, if any of your clients needed some archived data, would it be feasible to come back to the tapes, read, upload and share them? If you had a NAS and a fast enough internet connection, you may be able to host a site yourself, thus no need for reading the tape and uploading to a cloud.
Also, if it’s video footage, then you shouldn’t really count on LTO’s compression ability. It’s not particularly good for pictures and videos.
OP, look into AWS S3 Glacier. Not the normal S3.
In AUD, for a data centre in Sydney, with S3 Glacier Flexible you’re paying around $0.0045 per GB, and with Glacier Deep Archive it’s $0.002 per GB. This is your solution