Slashdot Mirror
The Future of the Cloud Depends On Magnetic Tape (arstechnica.com)
An anonymous reader quotes a report from Bloomberg: Although the century-old technology has disappeared from most people's daily view, magnetic tape lives on as the preferred medium for safely archiving critical cloud data in case, say, a software bug deletes thousands of Gmail messages, or a natural disaster wipes out some hard drives. The world's electronic financial, health, and scientific records, collected on state-of-the-art cloud servers belonging to Amazon.com, Microsoft, Google, and others, are also typically recorded on tape around the same time they are created. Usually the companies keep one copy of each tape on-site, in a massive vault, and send a second copy to somebody like Iron Mountain. Unfortunately for the big tech companies, the number of tape manufacturers has shrunk over the past three years from six to just two -- Sony and Fujifilm -- and each seems to think that's still one too many.
The Japanese companies have said the tape business is a mere rounding error as far as they're concerned, but each has spent millions of dollars arguing before the U.S. International Trade Commission to try to ban the other from importing tapes to America. [...] The tech industry worries that if Sony or Fujifilm knocks the other out of the U.S., the winner will hike prices, meaning higher costs for the big cloud providers; for old-line storage makers, including IBM, HPE, and Quantum; and, ultimately, for all those companies' customers. [...] Although Sony and Fujifilm have each assured the trade commission that they could fill the gap if their rival's products were shut out of the U.S., the need for storage continues to grow well beyond old conceptions. Construction is slated to begin as soon as next year on the Square Kilometer Array, a radio telescope with thousands of antennas in South Africa and Australia meant to detect signals emitted more than 13 billion years ago. It's been estimated the project could generate an exabyte (1 billion gigabytes) of raw data every day, the equivalent of 300 times the material in the U.S. Library of Congress and a huge storage headache all by itself.
The Japanese companies have said the tape business is a mere rounding error as far as they're concerned, but each has spent millions of dollars arguing before the U.S. International Trade Commission to try to ban the other from importing tapes to America. [...] The tech industry worries that if Sony or Fujifilm knocks the other out of the U.S., the winner will hike prices, meaning higher costs for the big cloud providers; for old-line storage makers, including IBM, HPE, and Quantum; and, ultimately, for all those companies' customers. [...] Although Sony and Fujifilm have each assured the trade commission that they could fill the gap if their rival's products were shut out of the U.S., the need for storage continues to grow well beyond old conceptions. Construction is slated to begin as soon as next year on the Square Kilometer Array, a radio telescope with thousands of antennas in South Africa and Australia meant to detect signals emitted more than 13 billion years ago. It's been estimated the project could generate an exabyte (1 billion gigabytes) of raw data every day, the equivalent of 300 times the material in the U.S. Library of Congress and a huge storage headache all by itself.
You've got to shave a lot of Japanese to get enough material for magnetic tape obviously, it's a scaling issue.
Not as easy as your think. The startup cost would be enormous. Very few engineers know, in detail, about thin film technology, it's kinda a lost art. just ask Kodak . The equipment would have to be custom made, no one has manufactured them in decades and the old one have long since been hauled off to the scrap yard.
Another day closer to redwood heaven
if the scenario described in the article happens and only one tape player will survive and prizes will go up, this will accelerate the death of tape storage. It seems that currently tape is still 2-3 times cheaper. It seems only a matter of time until tape will no more be competitive. There is still the legacy issue. Also, tape seems to last 30-50 years. It will be interesting to see whether a hard drive from today will still start up in 30 years. Officially, one estimates 10 years (but I guess it is more as I have been able to boot up drives older than 10 years). It will be important in the future to have cheap long term storage which lasts.
..truck loaded full of magnetic tapes.
Go ahead. Let us know how it goes.
Hint: you have to be able to produce thin plastic ribbons (5.6 micrometres thick for LTO-7 and LTO-8) that are close to a kilometre long. They need to be 12.65mm (plus or minus .006 mm) wide. You then need to bind barium ferrite particles to those ribbons, in a uniform pattern, to be able to hold 6,656 (LTO-8) tracks in that width, with a linear density of 20,668 bits in every mm (per track). And the ribbon needs to be able to stand up to at least 20,000 end-to-end passes.
This is not a trivial problem, and finding people who have a head start on solving it who don't already work for one of those manufacturers will be... difficult.
Construction is slated to begin as soon as next year on the Square Kilometer Array, a radio telescope with thousands of antennas in South Africa and Australia meant to detect signals emitted more than 13 billion years ago. It's been estimated the project could generate an exabyte (1 billion gigabytes) of raw data every day, the equivalent of 300 times the material in the U.S. Library of Congress and a huge storage headache all by itself
Good thing we just had a Slashdot article about intelligent compression. Even though most poo pooed it as not needed.
Shai Schticks:"You don't make peace with friends, you make peace with enemies"
I was talking with a coworker still at a former employer. He said their tape budget was up to $100k/month, that they were doing nearly a petabyte every night in backups. This company is still around and grown bigger since. They're not as big as Google, but they're up there.
Does not surprise me that this is still big business.
the project could generate an exabyte (1 billion gigabytes) of raw data every day, the equivalent of 300 LoCs.
Bytes, meters, inches, gallons, tons, carets, troy ounces, femtoseconds, microwatts -- pshaw, I was WONDERING when we were going to get back to normal units of measurement. Now who wants a pony?
Back on Topic: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. -- A. S. Tanenbaum
If the universe is someone's simulation -- does that mean the stars are just stuck pixels?
Profitable isn't enough, this is probably why so many closed down (without much warning! four in 3 years and I don't remember any single news article)
Naively, something with a permanent 0.1% profit would be excellent, meaning all wages and taxes are paid forever, etc. But well, there's the whole financial economy and all that. So, 3% to 5% for an industry should do? That was the norm before the neoliberals took over everything. Better sell everything, set on fire the now worthless tape fabrication equipment and run away with the money (to be invested for 10%, 15%)
Apparently a lone country that has its own ideas on how to do things can still do that, so Japan is left making the tapes. Maybe South Korea is another country, for the DRAM, flash memory, LCD. These are other examples of stuff nobody wants to make. How come they didn't sell out everything? Maybe their system with the grave corruption problem is not so bad.
Is there any particular reason why nobody makes a product that's basically like non-LTH (phase-change/magneto-optical) BD-R, but on a flexible film substrate stored on reels instead of bulky discs?
The main problem I see with magnetic tape is that it's inherently susceptible to stray magnetic fields (including the Earth's poles). In contrast, phase-change media can theoretically have a passive lifespan that's measured in decades (centuries, if "being able to read it with normal, consumer-grade hardware as a normal OS filesystem" isn't a hard requirement, and you can deal with forensic data-recovery using exotic purpose-built hardware).
You mean it can't be 3D printed?
Mostly random stuff.
transfer bandwidth is something quite different from job complete time
by the time you write to all those tapes the Square Kilometer Array will have generated more data...
they wont be using tapes for transfers...
3D? But you don't need it!
It's film, so it's 2D what we are talking here: a full 1D of net profit!
Parallel file store onto striped tape systems provides the best throughput for serial data.
Scientific data can be stored using parallel NetCDF, which is designed for such cases.
Tape is useless for random access, but that's not what you do with backup/restore or simple data logging from scientific instruments.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
Not as easy as your think. The startup cost would be enormous. Very few engineers know, in detail, about thin film technology, it's kinda a lost art. just ask Kodak . The equipment would have to be custom made, no one has manufactured them in decades and the old one have long since been hauled off to the scrap yard.
ATR Magnetics actually did this. They had their own coating machinery made. That's studio recording tape, though, so the tolerances will probably be a lot lower than for ultra-high density digital media on extremely thin backing.
What could ever replace the durabiliy of magnetic tape? Duct tape, maybe.
How obvious it is that a monopoly on magnetic tape is a bad thing, and yet the USA allow monopolies over all kinds of things involving last mile infrastructure and other critical services. But this one costs the tech companies so it's OK for the government to intervene but anywhere else and "IT'S TOO MUCH REGULATION".
>meaning higher costs for the big cloud providers
What part of the cost of the cloud-keeping is tape cost?
I do not believe in karma. "Funny"=-6. Do good and forbid evil. Yours, Oft-Offtopic Flamebaiting Troll.
You know backup your cloud to another cloud? Who needs this old fangled tape tech. LOL!
This just makes me remember at some point in the 90s when I was struggling with my 1GB hard drive and wanted more storage space. I'd heard about tape drives and DESPERATELY wanted one. I knew nothing more than they could hold a lot of stuff. Would be cool to just keep around as a memory these days.
I'm pretty sure I was under the impression that I could install computer games and such to it at the time, but I'm not sure if that's what I actually thought.
Go ahead. Let us know how it goes.
Hint: you have to be able to produce thin plastic ribbons (5.6 micrometres thick for LTO-7 and LTO-8) that are close to a kilometre long. They need to be 12.65mm (plus or minus .006 mm) wide. You then need to bind barium ferrite particles to those ribbons, in a uniform pattern, to be able to hold 6,656 (LTO-8) tracks in that width, with a linear density of 20,668 bits in every mm (per track).
True, but you only really need to worry about coating, slitting and polishing. My understanding from audio tape is that you usually buy in the backing from Dupont or someone. It's still not a trivial process, but it's not impossible either.
We must apply blockchain to this!
The only advantage magnetic tape has is it can hold more data.
Classic Slashdot
0.1% profit sounds terrible.
I'd want to at least beat inflation by a 1%.
a 2% TIPS being available would be the minimum return anyone with money would invest in I'd think.
Sure, if one is at 0.1% including depreciation, it doesn't make sense to close down, but it doesn't really make for a market one wants to invest in either.
Wow, sent an e-mail as suggested when clicking on "use classic" banner, and got a fast response that addressed my msg
Get congress to act. If they can reduce the size of their library to, say,500Mbytes, then 300 LOCs will fit on a thumb drive. Problem solved.
Nullius in verba
Not as easy as your think. The startup cost would be enormous. Very few engineers know, in detail, about thin film technology, it's kinda a lost art. just ask Kodak . The equipment would have to be custom made, no one has manufactured them in decades and the old one have long since been hauled off to the scrap yard.
ATR Magnetics actually did this. They had their own coating machinery made. That's studio recording tape, though, so the tolerances will probably be a lot lower than for ultra-high density digital media on extremely thin backing.
ATR seems to manufacture audio recording tape, not archival data storage tape. Magnetic storage drives run much faster than audio recording ones and they cannot be spliced if they break. Thus the specs and manufacturing costs are much higher. To be profitable a new company would have to sign up major cloud storage companies. These companies prioritize reliability based on "track record." A catch-22 situation for a startup with no history.
The problem is a number of things:
1: VCs want flash and sizzle. Tape is pedestrian, as opposed to some device that does little other than get compromised and send analytics back.
2: VCs want cheap. You cannot go the cheap route with tape.
3: There is a multi-trillion dollar push to get people to "the cloud". Tape goes against this.
4: There are many patents for density items.
5: There isn't a market for tape. Consumers don't care about backups, and businesses consider backups having no ROI for the most part.
6: A high capacity drive needs high I/O, and has to be done constantly to avoid "shoe-shining". You are not going to get a tape drive that plugs into USB that would be manageable, unless the drive was combined with a secondary media cache (SSD, hard disk) for staging.
7: A tape needs to be reliable. Finding engineers who have a mindset for designing for the long haul, as opposed to shipping a prototype out and letting legal handle the lawsuits is a mindset very uncommon in modern computing.
8: A tape has to have modern day features, like LTFS, encryption, and compression. Those are not easy to make or come by.
9: A tape needs to be able to be made from now to 10-20 years from now. Same with the drive. Maybe even 40 years.
10: There is already a market leader, LTO.
Now, what would be nice, would be an optical format. Sony has multi-terabyte formats. Optical has a lot of advantages over tape:
1: With antediluvian technologies (Burn-proof, etc.) buffer underruns are not an issue. So, having an optical drive hooked up to a smartphone can be doable for backups.
2: Optical is "cooler" than tape. The average consumer "gets" an optical drive, because CD/DVD/Blu Ray drives are common. So, Joe Sixpack will actually buy an optical drive for backups, given enough advertising.
3: Optical has a wider market.
4: Optical, in theory, can last a lot longer than magnetic tape, especially with inorganic technologies.
The problem with optical is that there has been no research on it in the past few years. It could easily compare to tape as a backup/storage medium, but the will isn't there for it to be improved.
It's completely unclear the BD-R optical discs consumers can get their hands on are superior, let alone more durable. CMC seems to currently be the least worst manufacturer, all the higher quality manufacturers have stopped making them, and they're obviously not highly trusted because of their history with previous generations of optical media.
For DVDs, I'd go with MAM-A, silver or gold, ditto CD-Rs, which I trust a lot more than DVD recordable media, since pressed DVDs pushed red laser CD technology as far as possible. (Taiyo Yuden exited the optical disc market in 2015, selling their stuff to CMC.) Therefore not going to calculate their costs, especially since their small capacity will start to really affect your off-site storage costs, unless you can stash them with friends or family, and trust them to keep the environment in which they're stored within the requirements (both tape and optical discs are picky here, that's the one advantage hard disks have over them.
Now we get to capacities, if you're going for low costs, single layer is where it's at, 25GB for BD-R, 4.7GB for DVD-R. Compare to 800GB native for LTO-4 tape, back when they were not ancient you could get new high quality Fujifilm ones for ~$22 in lots of 20, I now see a price of $14.70. I see today that Newegg is selling LTO-5 1.5TB native quantity 1 at $23, LTO-6 2.5TB native at $32, and LTO-7 6TB native at $82 (that's less than $1/TB more expensive than LTO-6), and a quick check at Amazon shows their LTO-6 and -7 prices are not competitive, even before we get into quantity discounts, which are the standard way to buy tape.
Comparing my first purchase of 25 Verbatim 25GB BD-Rs just this month from Amazon, to a quantity 20 price from a 3rd party I trust, Malelo and Company, for LTO-6 tape, we're talking $0.0352/GB vs. $0.0105/GB. LTO-5 weights in at $0.013/GB and LTO-4 at $0.0184. And I trust tape from Fujifilm infinitely more than I trust BD-Rs from CMC. Ah, Verbatim at quantity 50 BD-Rs only gets you down to $0.0306/GB.
Even grocery stores make more than that. I'd say a 3-10% profit margin would be useful.
Maybe do like gas stations -- have the media and drives be the low profit item, and then make the money on software or options. For example, the tape drives would ship with basic compression, password-based encryption, and crypto signing for WORM media. However, the tape drives could have optional licenses for more toys, such as better compression, deduplication, automatic expiration of media (so after a certain day, the tape cannot be read), and many other things.
https://www.nationalaudiocompa... Business is BOOMING. Not sure if they are into the mag tape storage/computer stuff, but boy do they churn out cassette & reel to reel tape.
Even though we haven't found anything better than tape yet.
http://progressquest.com/spoltog.php?name=Son+Of+Son+Of+DarkRookie
The real problem with optical is that for the consumer market, there's insufficient demand to keep quality BD-R manufacturers in business, per my research this month, CMC is the least worst manufacturer today (!!!), I'm not sure any BD-R media below CMC's level is going to last even a year.... MAM-A is still making I assume quality CD-R and DVD-R media and today they're the only company I'd trust for those formats. E.g. Taiyo Yuden exited that business and sold their stuff to CMC.
Tape, well, even if Sony is shut out of the US market, there's still the rest of the world for them to sell to. We'll see what happens, I trust Fujifilm somewhat more, but I won't enjoy paying monopoly prices to them. I agree there's no chance the Bay area VC community would even contemplate trying to set up a new LTO tape manufacturing company, but maybe one of the 4 companies that dropped out might restart, if they think they can be competitive while avoiding or licencing, if possible, Fujifilm's patents. Doubt it, though, there's likely good reasons they dropped out.
A few years back, I upgraded my storage server, new CPU, RAM, and a half dozen 4TB drives in a raidz2. Went from a few hundred gigs on failing old drives to 15 TBs. I had more space than I knew what to do with, so I started trying to bring everything I had burned to CD and DVD back online.
My old CDs from the 90s worked ok (yay Verbatim and old school TDK!) and I was able to read almost all of them after some cleanings. The DVD-R's were a different matter entirely. After as little as 5 years, they had degraded to the point where they were about 70% readable. No amount of cleaning or swapping out drives made any difference. Fortunately, this was mainly downloaded movies and music that could usually be easily replaced, so no big loss. As it turns out, consolidation of that market led to a race to the bottom in terms of ink quality and all of our burned DVD-Rs and BluRays are doomed to fail, unless we paid for pricey "archival quality" ones. I didn't.
Contrast this to work. At a former job, I recall digging a long retired LTO drive (an original one) out of storage in the basement and sticking a tape that had sat on a shelf for 10 years in it, and being able to pull files with no issues at all. Those things just work.
DVD-R technology is very marginal, the original pressed DVD version pushed red laser technology as far as it could go. But you still have to buy quality media, for CD-Rs I went with Taiyo Yuden (early on branded as Fujifilm in the US) and not a single one has failed me yet.
For DVD-R I got some from Taiyo Yuden as well, but didn't trust them if for no other reason than that they only cost a cent more than their CD-Rs. MAM-A gold DVD+Rs were my target for "archival quality", ought to test the few that I cut, but they were exposed to bad environmental conditions for a week or more so if they fail that won't tell me anything (note that Taiyo Yuden exited the optical disc business in 2015, selling their stuff to CMC).
Per my readings this month, comsumer BD-Rs are now a disaster, with CMC making the least worst (!!!). No joy there, and as you say, tape just works, still using the HP LTO-4 tape drive I bought in 2011 for ~$1,000 (it doesn't get a fraction of the wear a business would like put it to). Bought 60 tapes of Fujifilm and HP in quantity 2012, 2013 and 2014, plus some odd lots of those brands, and they're still doing fine.
Whatever happened to the holographic media that was suppose to kick tape to the curb?
As a consumer, for archiving, I use M-Disc in quad-layer Blu-Ray, which is 100 Gb per disc. Verbatim makes the discs I buy, and the drive was less than $100.
Learning HOW to think is more important than learning WHAT to think.
A unit of measurement that requires calculus to translate. Since the amount of data in an LoC is continuously changing by a variable amount of delta... I mean is there an LoC service I can ping like NTP to get the current value of its scale? Can I get access to historical data? I'm trying to read an article from 1992 and am having trouble understanding the scale it's trying to convey in 1992 LoCs!
I really need to sort out the conversion algorithms to other useful units of measure like:
LHCDO (Large Hadron Collider Data Output)
GWC (Google's Web Cache)
Parsecs (Because why aren't we measuring storage in Parsecs?? If it's good enough for the Kessel Run it's good enough for storage needs!)
10 years ago we backed up using tapes. Always always always had issues with reliability of the data on them. Talking to peers, everyone experienced these problems. I don't see anybody talking about it here... am I missing something?
Are you sure there's any original M-Disc technology in those M-Disc Verbatim labeled BD-XL triple layer discs? The parent company, which made its questionable name in single layer DVD media (inner lot variability was awful), went bankrupt in December 2016, sounds like Chapter 7 where the creditors got all the assets and set up a company named Yours.co to sell discs and such. The first 2 layers have to be somewhat transparent, right?
Tape drive prices are all over the place, there's lots of variables there including of course speed and generation, but you're completely off about media. Amazon prices from a good 3rd party company I've done business with before, Fujifilm tapes, for LTO-6, we're talking $0.0352/GB, LTO-5 weights in at $0.013/GB and LTO-4 at $0.0184.
Newegg prices for 8TB drives, the current sweet spot in capacity, range from $0.0256/GB to $0.0325 for the lower end of the 5 year warranty 550TB bandwidth/year enterprise drives. I hope I don't have to look up per GB prices for the higher capacity drives, companies by them and the higher capacity LTO tapes because space for them isn't cheap.
The LTO drives do cost 10x a hard drive, but you can put cheap media in them, and all day, your claim they are only meant to run a few hundred tapes before breaking, and the tapes are "even much less reliable" is falsified by my own experience with an HP LTO-4 drive and 65 Fujifilm and "HP" tapes over the last 6 years, incremental backups every day, full 3-5 tape backups once a month (and was doing Bacula differentials every week for some years).
Enterprises use them because they're very reliable, less fragile than hard drives which makes them good for off-site storage, etc. Cloud companies, which this article used as a hook since they're currently sexy? Only for archival storage, maybe, like AWS Glacier and Azure Archive, obviously any class of storage that's available within less than a second isn't going to be on tape. See also all the scientific uses where 100s of petabytes which don't have to be frequently accessed are reliably stored on LTO tape (you think they're willing to lose data and thus papers and their careers???).
Correction, for LTO-6 tape we're talking $0.0105/GB, that $0.0352/GB price was for Verbatim single layer BD-R disks in quantity 25.
I am confused. The 100-GB Blu-ray disks that you buy cost about $20, which is much more expensive per GB than the cost of a 2.5" external drive. This is without counting the cost of your Blu-ray drive, the hassle of keeping around many more disks than hard drives (20 of your disks for a single 2 TB drive, for example), or the much lower read/write speed of the Blu-ray. And if portability is not an issue, you can get an even cheaper 3.5" external disk.
They're archives, so speed doesn't really mean much in the equation and the actual cost is around $15 per disc in lots of 5 from Amazon. My main focus is reliability, for this use. I can burn data to the disk, put it in a jewel case, and set it on a closet shelf at my parent's house and forget about it.
Honestly, the vast majority of consumers don't have 100 Gb of data they need to back up anyway. I only use 1 disc for family pictures, tax returns, non-media storage, and that is overkill. The rest is a back up of all my ripped music and video media. I just don't want to go thru the hassle of ripping hundreds of CDs, DVDs, and Blu-Rays for my digital collection again.
I don't trust a 2.5" external drive to retain the data for years. That isn't archival media.
Learning HOW to think is more important than learning WHAT to think.