Slashdot Mirror
The Future of the Cloud Depends On Magnetic Tape (arstechnica.com)
An anonymous reader quotes a report from Bloomberg: Although the century-old technology has disappeared from most people's daily view, magnetic tape lives on as the preferred medium for safely archiving critical cloud data in case, say, a software bug deletes thousands of Gmail messages, or a natural disaster wipes out some hard drives. The world's electronic financial, health, and scientific records, collected on state-of-the-art cloud servers belonging to Amazon.com, Microsoft, Google, and others, are also typically recorded on tape around the same time they are created. Usually the companies keep one copy of each tape on-site, in a massive vault, and send a second copy to somebody like Iron Mountain. Unfortunately for the big tech companies, the number of tape manufacturers has shrunk over the past three years from six to just two -- Sony and Fujifilm -- and each seems to think that's still one too many.
The Japanese companies have said the tape business is a mere rounding error as far as they're concerned, but each has spent millions of dollars arguing before the U.S. International Trade Commission to try to ban the other from importing tapes to America. [...] The tech industry worries that if Sony or Fujifilm knocks the other out of the U.S., the winner will hike prices, meaning higher costs for the big cloud providers; for old-line storage makers, including IBM, HPE, and Quantum; and, ultimately, for all those companies' customers. [...] Although Sony and Fujifilm have each assured the trade commission that they could fill the gap if their rival's products were shut out of the U.S., the need for storage continues to grow well beyond old conceptions. Construction is slated to begin as soon as next year on the Square Kilometer Array, a radio telescope with thousands of antennas in South Africa and Australia meant to detect signals emitted more than 13 billion years ago. It's been estimated the project could generate an exabyte (1 billion gigabytes) of raw data every day, the equivalent of 300 times the material in the U.S. Library of Congress and a huge storage headache all by itself.
The Japanese companies have said the tape business is a mere rounding error as far as they're concerned, but each has spent millions of dollars arguing before the U.S. International Trade Commission to try to ban the other from importing tapes to America. [...] The tech industry worries that if Sony or Fujifilm knocks the other out of the U.S., the winner will hike prices, meaning higher costs for the big cloud providers; for old-line storage makers, including IBM, HPE, and Quantum; and, ultimately, for all those companies' customers. [...] Although Sony and Fujifilm have each assured the trade commission that they could fill the gap if their rival's products were shut out of the U.S., the need for storage continues to grow well beyond old conceptions. Construction is slated to begin as soon as next year on the Square Kilometer Array, a radio telescope with thousands of antennas in South Africa and Australia meant to detect signals emitted more than 13 billion years ago. It's been estimated the project could generate an exabyte (1 billion gigabytes) of raw data every day, the equivalent of 300 times the material in the U.S. Library of Congress and a huge storage headache all by itself.
Not as easy as your think. The startup cost would be enormous. Very few engineers know, in detail, about thin film technology, it's kinda a lost art. just ask Kodak . The equipment would have to be custom made, no one has manufactured them in decades and the old one have long since been hauled off to the scrap yard.
Another day closer to redwood heaven
I was talking with a coworker still at a former employer. He said their tape budget was up to $100k/month, that they were doing nearly a petabyte every night in backups. This company is still around and grown bigger since. They're not as big as Google, but they're up there.
Does not surprise me that this is still big business.
There's another factor you didn't consider that limits tape usage to very high volume use only. The tape is cheap, but the drives themselves certainly are not - for your LTO8 tape you need a £3000 drive, plus a server with its own RAID storage array to maintain the required transfer rate and an SAS controller. That's why tape completely disappeared from consumer use and almost disappeared from SOHO - you need to be thinking in terms of multiple hundreds of terabytes to justify the initial equipment costs, even if the tapes themselves are affordable.
"Also I sure as hell wouldn't want to rely on a HDD spinning up for the first time after sitting on a shelf for 10 years, let alone 20 or 30 years. Sure it might, but I wouldn't want to rely on it."
That's again, a matter of costs. Tapes have a low marginal cost, which is their "natural" advantage. Hard disks are cheap because sheer volume, but their marginal costs must be higher, so they are cheap... as long as you can use off-the-shelve disks for your own purposes.
What I mean is, in this case, that nowhere says a hard disk must come with its own spinning engine. It's not that hard to think about a "hard disk" to be mounted on an spinning machine (that's basically what you do with tapes) so, on one hand, you cut per-unit costs (only one "spinning machine" for thousands of "disks") and increase reliability (having one engine per thousands of disks allows for building it sturdy and you use it daily instead of once in ten years). Problem? the same than tapes: unless you manage to make this "spinless hard disk" the industry standard, it becomes a niche application, which increases costs.
By the way, there isn't already a standard industry for spinless hard disks? solid state, or something like that?
How obvious it is that a monopoly on magnetic tape is a bad thing, and yet the USA allow monopolies over all kinds of things involving last mile infrastructure and other critical services. But this one costs the tech companies so it's OK for the government to intervene but anywhere else and "IT'S TOO MUCH REGULATION".
These tapes are in massive libraries. Typically you string ~16 rack sized chassis together, which using LTO8 will give you around ~300PB of uncompressed storage. We are talking the likes of the IBM TS3500 or the newer TS4500. The other two players in the market are SpectraLogic and Oracle/StorTek who have similar libraries. With the TS3500 you used to be able at least to get a passover option so you could string 15 rows of libraries together for insane amounts of storage.
The only thing I doubt in the article is the rubbish about Iron Mountain. If you are Google/Facebook/Amazon etc. you just replicate your data to one of your many remote data centers. No point messing about having humans physically handling tapes on a daily basis (well other than feeding Audrey with new tapes). To be honest actually having your tape library onsite is a fairly dumb tactic anyway as there is a good chance the reason you need to use your backup is because the data centre has been transformed into a pile of smoking rubble or a large swimming pool.
Finally when you need to change tape technologies you just have the software copy it from the old tapes to the new tapes, and if they are all in the library while it might take several months it does not involve human interaction. Other than take old tapes out and putting new ones in every few days. You are going to need to do this probably every 6-7 years if you stretch it so the tapes only need to be able to reliably store data for say 10 years. That said provided it is stored in the correct environment LTO is good for 30 years from memory.
If you're a prosumer like myself, who's been personally using magnetic tape since the late 1970s, you don't buy the very latest generation of LTO, you go back one or more generations. If you're good at scrounging, you should be able to get a tape drive with a lot of life for $1,000 or less, as I did in 2011 for a new HP LTO-4 drive which I'm still using, when LTO-5 was the new hotness. Today, without going to any real effort, I could buy a new LTO-5 drive from Newegg for $1,700.
A SAS controller is pretty cheap, ones based on the LSI200x chip particularly so right now. If you're patient, you only need a buffer device to avoid shoe-shining your tapes, rather than a fast RAID array. That is, you have your backup software write a 10GiB or so file at a time (probably ought to be larger for the newer LTO generations), and then stream it to the tape drive in one go. I use a Cheetah 15K drive right now, a DRAM tmp file would suffice, you can also scrounge higher end used but not used up Intel DC for enterprise datacenter 2.5 inch drives that were presumably taken out of service because their "small" capacity could no longer be justified.
All in all its quite practical, you have to do the math, including safe deposit box rental or whatever you choose for your offsite storage, and see if it makes sense compared to cloud vendors. Or you might not trust them to exclusively do your offsite backups, tape gives you a level of control you don't get from them.
It's completely unclear the BD-R optical discs consumers can get their hands on are superior, let alone more durable. CMC seems to currently be the least worst manufacturer, all the higher quality manufacturers have stopped making them, and they're obviously not highly trusted because of their history with previous generations of optical media.
For DVDs, I'd go with MAM-A, silver or gold, ditto CD-Rs, which I trust a lot more than DVD recordable media, since pressed DVDs pushed red laser CD technology as far as possible. (Taiyo Yuden exited the optical disc market in 2015, selling their stuff to CMC.) Therefore not going to calculate their costs, especially since their small capacity will start to really affect your off-site storage costs, unless you can stash them with friends or family, and trust them to keep the environment in which they're stored within the requirements (both tape and optical discs are picky here, that's the one advantage hard disks have over them.
Now we get to capacities, if you're going for low costs, single layer is where it's at, 25GB for BD-R, 4.7GB for DVD-R. Compare to 800GB native for LTO-4 tape, back when they were not ancient you could get new high quality Fujifilm ones for ~$22 in lots of 20, I now see a price of $14.70. I see today that Newegg is selling LTO-5 1.5TB native quantity 1 at $23, LTO-6 2.5TB native at $32, and LTO-7 6TB native at $82 (that's less than $1/TB more expensive than LTO-6), and a quick check at Amazon shows their LTO-6 and -7 prices are not competitive, even before we get into quantity discounts, which are the standard way to buy tape.
Comparing my first purchase of 25 Verbatim 25GB BD-Rs just this month from Amazon, to a quantity 20 price from a 3rd party I trust, Malelo and Company, for LTO-6 tape, we're talking $0.0352/GB vs. $0.0105/GB. LTO-5 weights in at $0.013/GB and LTO-4 at $0.0184. And I trust tape from Fujifilm infinitely more than I trust BD-Rs from CMC. Ah, Verbatim at quantity 50 BD-Rs only gets you down to $0.0306/GB.