The Future of the Cloud Depends On Magnetic Tape (arstechnica.com)

An anonymous reader quotes a report from Bloomberg: Although the century-old technology has disappeared from most people's daily view, magnetic tape lives on as the preferred medium for safely archiving critical cloud data in case, say, a software bug deletes thousands of Gmail messages, or a natural disaster wipes out some hard drives. The world's electronic financial, health, and scientific records, collected on state-of-the-art cloud servers belonging to Amazon.com, Microsoft, Google, and others, are also typically recorded on tape around the same time they are created. Usually the companies keep one copy of each tape on-site, in a massive vault, and send a second copy to somebody like Iron Mountain. Unfortunately for the big tech companies, the number of tape manufacturers has shrunk over the past three years from six to just two -- Sony and Fujifilm -- and each seems to think that's still one too many.

The Japanese companies have said the tape business is a mere rounding error as far as they're concerned, but each has spent millions of dollars arguing before the U.S. International Trade Commission to try to ban the other from importing tapes to America. [...] The tech industry worries that if Sony or Fujifilm knocks the other out of the U.S., the winner will hike prices, meaning higher costs for the big cloud providers; for old-line storage makers, including IBM, HPE, and Quantum; and, ultimately, for all those companies' customers. [...] Although Sony and Fujifilm have each assured the trade commission that they could fill the gap if their rival's products were shut out of the U.S., the need for storage continues to grow well beyond old conceptions. Construction is slated to begin as soon as next year on the Square Kilometer Array, a radio telescope with thousands of antennas in South Africa and Australia meant to detect signals emitted more than 13 billion years ago. It's been estimated the project could generate an exabyte (1 billion gigabytes) of raw data every day, the equivalent of 300 times the material in the U.S. Library of Congress and a huge storage headache all by itself.

Comments

So why doesn't somebody

[rsilvergun]

just start a tape company. I mean, with so little competition wouldn't it be instantly profitable?

Re:So why doesn't somebody

[Anonymous Coward]

You've got to shave a lot of Japanese to get enough material for magnetic tape obviously, it's a scaling issue.

Re:So why doesn't somebody

[Ziest]

Not as easy as your think. The startup cost would be enormous. Very few engineers know, in detail, about thin film technology, it's kinda a lost art. just ask Kodak . The equipment would have to be custom made, no one has manufactured them in decades and the old one have long since been hauled off to the scrap yard.

Re:So why doesn't somebody

[Anonymous Coward]

Go ahead. Let us know how it goes.

Hint: you have to be able to produce thin plastic ribbons (5.6 micrometres thick for LTO-7 and LTO-8) that are close to a kilometre long. They need to be 12.65mm (plus or minus .006 mm) wide. You then need to bind barium ferrite particles to those ribbons, in a uniform pattern, to be able to hold 6,656 (LTO-8) tracks in that width, with a linear density of 20,668 bits in every mm (per track). And the ribbon needs to be able to stand up to at least 20,000 end-to-end passes.

This is not a trivial problem, and finding people who have a head start on solving it who don't already work for one of those manufacturers will be... difficult.

Re:So why doesn't somebody

[Anonymous Coward]

Profitable isn't enough, this is probably why so many closed down (without much warning! four in 3 years and I don't remember any single news article)
Naively, something with a permanent 0.1% profit would be excellent, meaning all wages and taxes are paid forever, etc. But well, there's the whole financial economy and all that. So, 3% to 5% for an industry should do? That was the norm before the neoliberals took over everything. Better sell everything, set on fire the now worthless tape fabrication equipment and run away with the money (to be invested for 10%, 15%)

Apparently a lone country that has its own ideas on how to do things can still do that, so Japan is left making the tapes. Maybe South Korea is another country, for the DRAM, flash memory, LCD. These are other examples of stuff nobody wants to make. How come they didn't sell out everything? Maybe their system with the grave corruption problem is not so bad.

Re:So why doesn't somebody

[50000BTU_barbecue]

You mean it can't be 3D printed?

Re:So why doesn't somebody

[Anonymous Coward]

Why 3D print? Replace a tape collection with a million raspberry Pi and you're done.

Re:So why doesn't somebody

[Anonymous Coward]

Well maybe not now, but surely in 10 years. Look how far microprocessors came in such a short time.

Re:So why doesn't somebody

[turbidostato]

3D? But you don't need it!

It's film, so it's 2D what we are talking here: a full 1D of net profit!

Re:So why doesn't somebody

[Tapewolf]

Not as easy as your think. The startup cost would be enormous. Very few engineers know, in detail, about thin film technology, it's kinda a lost art. just ask Kodak . The equipment would have to be custom made, no one has manufactured them in decades and the old one have long since been hauled off to the scrap yard.

ATR Magnetics actually did this. They had their own coating machinery made. That's studio recording tape, though, so the tolerances will probably be a lot lower than for ultra-high density digital media on extremely thin backing.

Re:So why doesn't somebody

[Tapewolf]

Go ahead. Let us know how it goes.

Hint: you have to be able to produce thin plastic ribbons (5.6 micrometres thick for LTO-7 and LTO-8) that are close to a kilometre long. They need to be 12.65mm (plus or minus .006 mm) wide. You then need to bind barium ferrite particles to those ribbons, in a uniform pattern, to be able to hold 6,656 (LTO-8) tracks in that width, with a linear density of 20,668 bits in every mm (per track).

True, but you only really need to worry about coating, slitting and polishing. My understanding from audio tape is that you usually buy in the backing from Dupont or someone. It's still not a trivial process, but it's not impossible either.

Re:So why doesn't somebody

[Anonymous Coward]

We must apply blockchain to this!

Re:So why doesn't somebody

[AvitarX]

0.1% profit sounds terrible.

I'd want to at least beat inflation by a 1%.

a 2% TIPS being available would be the minimum return anyone with money would invest in I'd think.

Sure, if one is at 0.1% including depreciation, it doesn't make sense to close down, but it doesn't really make for a market one wants to invest in either.

Re:So why doesn't somebody

[TaleWeaver]

Not as easy as your think. The startup cost would be enormous. Very few engineers know, in detail, about thin film technology, it's kinda a lost art. just ask Kodak . The equipment would have to be custom made, no one has manufactured them in decades and the old one have long since been hauled off to the scrap yard.

ATR Magnetics actually did this. They had their own coating machinery made. That's studio recording tape, though, so the tolerances will probably be a lot lower than for ultra-high density digital media on extremely thin backing.

ATR seems to manufacture audio recording tape, not archival data storage tape. Magnetic storage drives run much faster than audio recording ones and they cannot be spliced if they break. Thus the specs and manufacturing costs are much higher. To be profitable a new company would have to sign up major cloud storage companies. These companies prioritize reliability based on "track record." A catch-22 situation for a startup with no history.

Re:So why doesn't somebody

[ctilsie242]

The problem is a number of things:

1: VCs want flash and sizzle. Tape is pedestrian, as opposed to some device that does little other than get compromised and send analytics back.
2: VCs want cheap. You cannot go the cheap route with tape.
3: There is a multi-trillion dollar push to get people to "the cloud". Tape goes against this.
4: There are many patents for density items.
5: There isn't a market for tape. Consumers don't care about backups, and businesses consider backups having no ROI for the most part.
6: A high capacity drive needs high I/O, and has to be done constantly to avoid "shoe-shining". You are not going to get a tape drive that plugs into USB that would be manageable, unless the drive was combined with a secondary media cache (SSD, hard disk) for staging.
7: A tape needs to be reliable. Finding engineers who have a mindset for designing for the long haul, as opposed to shipping a prototype out and letting legal handle the lawsuits is a mindset very uncommon in modern computing.
8: A tape has to have modern day features, like LTFS, encryption, and compression. Those are not easy to make or come by.
9: A tape needs to be able to be made from now to 10-20 years from now. Same with the drive. Maybe even 40 years.
10: There is already a market leader, LTO.

Now, what would be nice, would be an optical format. Sony has multi-terabyte formats. Optical has a lot of advantages over tape:

1: With antediluvian technologies (Burn-proof, etc.) buffer underruns are not an issue. So, having an optical drive hooked up to a smartphone can be doable for backups.
2: Optical is "cooler" than tape. The average consumer "gets" an optical drive, because CD/DVD/Blu Ray drives are common. So, Joe Sixpack will actually buy an optical drive for backups, given enough advertising.
3: Optical has a wider market.
4: Optical, in theory, can last a lot longer than magnetic tape, especially with inorganic technologies.

The problem with optical is that there has been no research on it in the past few years. It could easily compare to tape as a backup/storage medium, but the will isn't there for it to be improved.

Re:So why doesn't somebody

[ctilsie242]

Even grocery stores make more than that. I'd say a 3-10% profit margin would be useful.

Maybe do like gas stations -- have the media and drives be the low profit item, and then make the money on software or options. For example, the tape drives would ship with basic compression, password-based encryption, and crypto signing for WORM media. However, the tape drives could have optional licenses for more toys, such as better compression, deduplication, automatic expiration of media (so after a certain day, the tape cannot be read), and many other things.

Re:So why doesn't somebody

[Anonymous Coward]

Because it's not?

The company I do work for uses a Quantum scalar i80, Iron Mountain comes in once a week and switches out the tapes. They switch them out two Fujifilm boxes at a time (20 tapes each.)

The issue isn't really the tape suppliers, but the quantity. If you switch tape brands, the lubrication gunks up, so basically any tape used, must be the sole brand used for the life of the tape drive. I believe we're using LTO-6 but I haven't actually looked in the boxes, I just know they're there, it's not my job.

So lets' say a Tsunami takes out Sony and Fujifilm for two weeks, well we're fucked then, because we only have 4 boxes in reserve, or two weeks worth.

Re:So why doesn't somebody

[mangastudent]

The real problem with optical is that for the consumer market, there's insufficient demand to keep quality BD-R manufacturers in business, per my research this month, CMC is the least worst manufacturer today (!!!), I'm not sure any BD-R media below CMC's level is going to last even a year.... MAM-A is still making I assume quality CD-R and DVD-R media and today they're the only company I'd trust for those formats. E.g. Taiyo Yuden exited that business and sold their stuff to CMC.

Tape, well, even if Sony is shut out of the US market, there's still the rest of the world for them to sell to. We'll see what happens, I trust Fujifilm somewhat more, but I won't enjoy paying monopoly prices to them. I agree there's no chance the Bay area VC community would even contemplate trying to set up a new LTO tape manufacturing company, but maybe one of the 4 companies that dropped out might restart, if they think they can be competitive while avoiding or licencing, if possible, Fujifilm's patents. Doubt it, though, there's likely good reasons they dropped out.

Re:So why doesn't somebody

[Anonymous Coward]

You have no idea what you're talking about.

Tape is tape, there has to be absolutely no bells and whistles, otherwise it's incompatible with other tapes and tape drives. Let's say in theory that my office catches fire and we lose the 50 tapes in the machine. Well in order to restore the tapes that are held by Iron Mountain, we'd have to buy the EXACT SAME model of tape drive/vault and server, which may be 10 years old at the time.

Basically the gist of the problem is maintaining a supply of tapes, not how many manufacturers there are. If Japan is swallowed in a Tsunami, and production has a two week gap, that is every cloud provider being fucked over for two weeks as well. If say Google has a fire in the mean time, they may lose 4 weeks of data. I need not remind everyone that Google's Youtube and Gmail services simply can't afford to do that.

Re:So why doesn't somebody

[chill]

As a consumer, for archiving, I use M-Disc in quad-layer Blu-Ray, which is 100 Gb per disc. Verbatim makes the discs I buy, and the drive was less than $100.

Re:So why doesn't somebody

[mangastudent]

Are you sure there's any original M-Disc technology in those M-Disc Verbatim labeled BD-XL triple layer discs? The parent company, which made its questionable name in single layer DVD media (inner lot variability was awful), went bankrupt in December 2016, sounds like Chapter 7 where the creditors got all the assets and set up a company named Yours.co to sell discs and such. The first 2 layers have to be somewhat transparent, right?

Re:So why doesn't somebody

[Anonymous Coward]

Thanks Captain Fucking Obvious. Colonel Can't Fucking Read sends a message stating "THAT'S WHAT OP FUCKING SAID".

Re:So why doesn't somebody

[orionbelt]

I am confused. The 100-GB Blu-ray disks that you buy cost about $20, which is much more expensive per GB than the cost of a 2.5" external drive. This is without counting the cost of your Blu-ray drive, the hassle of keeping around many more disks than hard drives (20 of your disks for a single 2 TB drive, for example), or the much lower read/write speed of the Blu-ray. And if portability is not an issue, you can get an even cheaper 3.5" external disk.

Re:So why doesn't somebody

[chill]

They're archives, so speed doesn't really mean much in the equation and the actual cost is around $15 per disc in lots of 5 from Amazon. My main focus is reliability, for this use. I can burn data to the disk, put it in a jewel case, and set it on a closet shelf at my parent's house and forget about it.

Honestly, the vast majority of consumers don't have 100 Gb of data they need to back up anyway. I only use 1 disc for family pictures, tax returns, non-media storage, and that is overkill. The rest is a back up of all my ripped music and video media. I just don't want to go thru the hassle of ripping hundreds of CDs, DVDs, and Blu-Rays for my digital collection again.

I don't trust a 2.5" external drive to retain the data for years. That isn't archival media.

Non-magnetic tape is where it's headed

[Anonymous Coward]

so buy now and get in while the getting is good.

long term solutions

[e**(i pi)-1]

if the scenario described in the article happens and only one tape player will survive and prizes will go up, this will accelerate the death of tape storage. It seems that currently tape is still 2-3 times cheaper. It seems only a matter of time until tape will no more be competitive. There is still the legacy issue. Also, tape seems to last 30-50 years. It will be interesting to see whether a hard drive from today will still start up in 30 years. Officially, one estimates 10 years (but I guess it is more as I have been able to boot up drives older than 10 years). It will be important in the future to have cheap long term storage which lasts.

Re:long term solutions

[Harlequin80]

$150ish for 30tb of storage using LTO 8. WD Red 10tb drives are ~400ish. So storage cost is more like 8 to 10 times cheaper. And thats enterprise grade vs "prosumer" grade.

On top of that you need 3 drives to equal 1 tape. So storage costs are 3 times higher. Also I sure as hell wouldn't want to rely on a HDD spinning up for the first time after sitting on a shelf for 10 years, let alone 20 or 30 years. Sure it might, but I wouldn't want to rely on it.

As far as I am aware tape is currently the only viable long term archival storage method.

Re:long term solutions

[Anonymous Coward]

You forgot to add in ~$4k for a LTO 8 Drive, using that same $4k would get you ~100TB worth of HDDs (10x 10TB @ $400).

Re:long term solutions

[dgatwood]

Also, LTO-8 has only 12 TB of native capacity. Ostensibly, you can get up to 30 TB of storage per tape, but that's the best-case scenario. Realistically, you need to assume that you'll get 12 TB per tape, and if you get more, yay.

Re:long term solutions

[SuricouRaven]

There's another factor you didn't consider that limits tape usage to very high volume use only. The tape is cheap, but the drives themselves certainly are not - for your LTO8 tape you need a £3000 drive, plus a server with its own RAID storage array to maintain the required transfer rate and an SAS controller. That's why tape completely disappeared from consumer use and almost disappeared from SOHO - you need to be thinking in terms of multiple hundreds of terabytes to justify the initial equipment costs, even if the tapes themselves are affordable.

Re:long term solutions

[dgatwood]

More to the point, $150 will buy you a basic 8 TB external hard drive these days. So in the worst-case compression scenario, tape is only about 2/3rds the price of storing the data on hard drives, assuming you have sufficient physical space to store the hard drives, and ignoring the cost of the tape drive. (More to the point, if compression can let you store more data on tape, that same compression would also let you store more on a hard drive, so for a fair comparison, only the native capacity matters.)

As soon as you add in the cost of the tape drive, you can compute a break-even point where tape becomes cheaper. The break-even point is where the cost of hard drives, x, equals the cost of tapes, which is the cost of the tape drive ($4k) plus 2/3 * x. So x = $4000 + .66666x. Solve for x, and you get x/3 = $4000, so x = $12000.

At $150 for 8 TB, that comes to 640 TB. Realistically, that ought to be enough for anybody. :-)

But I guess if you have to store more than that, tape might be a good deal.

*shrugs*

Re:long term solutions

[turbidostato]

"Also I sure as hell wouldn't want to rely on a HDD spinning up for the first time after sitting on a shelf for 10 years, let alone 20 or 30 years. Sure it might, but I wouldn't want to rely on it."

That's again, a matter of costs. Tapes have a low marginal cost, which is their "natural" advantage. Hard disks are cheap because sheer volume, but their marginal costs must be higher, so they are cheap... as long as you can use off-the-shelve disks for your own purposes.

What I mean is, in this case, that nowhere says a hard disk must come with its own spinning engine. It's not that hard to think about a "hard disk" to be mounted on an spinning machine (that's basically what you do with tapes) so, on one hand, you cut per-unit costs (only one "spinning machine" for thousands of "disks") and increase reliability (having one engine per thousands of disks allows for building it sturdy and you use it daily instead of once in ten years). Problem? the same than tapes: unless you manage to make this "spinless hard disk" the industry standard, it becomes a niche application, which increases costs.

By the way, there isn't already a standard industry for spinless hard disks? solid state, or something like that?

Re:long term solutions

[Anonymous Coward]

SSD storage might become a thing for the buffer to feed the tape drive(s).
You would get away with a single interim drive? One really big but with no advanced performance features and stuff.

Re:long term solutions

[Anonymous Coward]

But you are ignoring the fact that long term, the magnetic patterns on the tape tend to degrade each other. I would not trust tape that was stored that long. Magnetic tape may be the most economical for short term storage, but is not reliable for long term (20 years or more).

Re:long term solutions

[tbq]

What I mean is, in this case, that nowhere says a hard disk must come with its own spinning engine.

That's exactly what an Iomega Jaz disk was, with a hard drive platter in a removable cartridge while the motors and heads were in the drive itself.

Re:long term solutions

[Anonymous Coward]

2x... if your doing it for long term storage you need to store a second drive for the future person to read from....

Re:long term solutions

[Antique Geekmeister]

The "spinless hard disks" or SSD drives are much, much more expensive per Gigabyte than hard drives or tape.

Re:long term solutions

[AmiMoJo]

Or just pay someone else to do it. Chances are you can't beat services that do this on a massive scale, with wide geographic distribution and duplication, and 3-9s SLAs.

For example Microsoft charges $720/year for 30TB off archive storage, plus a few hundred for bandwidth. I doubt you could do it cheaper with tape, especially when you factor in periodic upgrades, duplication and operating costs.

Re:long term solutions

[Anonymous Coward]

Who might need 640 TB or more of storage? Let me see.

Video. 8K UHD: 7680x4320 pixels, or 33,177,600 pixels. HDR gives a bit depth of 10 bpp, for 40,500 kB per frame. 60 frames per second equals 2.3 GB per second. A two hour movie at that resolution, uncompressed, will be about 16 TB in size. Add in all the extra footage that will be shot, plus audio, and so forth, and it wouldn't take long for a movie (or TV!) studio to need multiple petabytes of storage. Yes, they can compress the data to a certain extent - but production studios won't use lossy compression until the final stages of production, when they're creating the media for sending out to theatres (or mastering the optical disc release, or similar), so the data reduction would probably be a factor of 50% at most.

Research. Medical research video easily gets up into the multiple terabytes, for the same reasons. I've had talks with a research institution that needs to be able to push multiple TB into AWS for computation, retrieving a relatively small output data set at the end (the source data is abandoned; they keep a copy locally.) All of that data needs to be kept so that the research can be validated later on if need be.

The Large Hadron Collider - about 25 GB/s incoming, although they do throw away a fair bit of that heuristically.

There are plenty of use cases out there that have massive data requirements. They're just not your typical at-home use cases (and also don't generally fit in with the needs of most businesses, either.) Tape is still damn hard to beat at that sort of scale. It's not the slam dunk that it used to be, no - but it's nowhere near dead. I guarantee you, services like Amazon Glacier, Google Cloud Storage, and Azure Standard Archive Storage will all have tape behind them, even if the companies aren't willing to admit it.

Re:long term solutions

[Xenx]

Except, they were discussing costs for being the Microsoft in your solution. It's kind of the whole point of the story.

Re: long term solutions

[Anonymous Coward]

The same is true for magnetic patterns on disc platters. Ditto the charge in SSDs, which decays over time unless refreshed. Not to mention Cd/DVD/Bluray optical storage which has problems with chemical degradation causing the backing to come away. Unless you're willing to chisel the bits by hand deeply into granite, I think you're going to be dissapointed with the longevity.

Re: long term solutions

[Harlequin80]

The point of the article is large scale storage for cloud. $4k on a drive is a rounding error in these scales.

When you are talking 1000s of tapes, and the humidity and temperature controlled archival storage costs the price of the drives is tiny.

Re: long term solutions

[Harlequin80]

Ive never had an issue getting the compressed data amounts onto lto 5 tapes. I dont say any reason that lto8 would suddenly be different.

That said im looking at this from a large scale commercial archival perspective. And in those cases the tapes go off site to an environmentally controlled store.

Then you look at the ease of use you get when you add in tape libraries. For example ones that hold 24 tapes you start to see why they beat other storages.

Re: long term solutions

[Harlequin80]

To date all lto drives have been backwards compatible.

Re:long term solutions

[AmiMoJo]

Really, dgatwood was comparing the price to Microsoft of a consumer USB 8TB hard drive to tape storage? And Harlequin80's concerns about a HDD spinning up after 10 years sitting on a shelf is something that they have to contend with in Microsoft datacentres?

Re:long term solutions

[Harlequin80]

For an enterprise the cost of the drives is the smallest part of the total cost.

If you are needing large scale long term archival storage you are using tape libraries anyway and they are far far more than 3000 pounds. Even then tape beats all other options currently.

Re:long term solutions

[Harlequin80]

Kinda depends on what your aim is. Always on storage then hard disks. But not for archival purposes.

One example of stuff I have worked on is the storage of seismic data. It is simply too much data to keep hot all the time. Especially since most of it is never used. So it is stored on tape. It goes into an environmentally controlled building and goes into a data migration programme where the tapes are read onto new media every 10 years.

I'm sure there is no technical reason you couldn't drop HDD platters into a reader, but the nice thing about LTO is they are all backwards compatible. An LTO 1 tape can be read by an LTO 8 drive.

Re:long term solutions

[Anonymous Coward]

SSDs are also much much less likely to survive 10 years off power without loosing data. Same for USB sticks and other forms of flash.

Re:long term solutions

[Anonymous Coward]

You could just look at what Engineers say on the topic:
Why the Future of Data Storage is (Still) Magnetic Tape

Re:long term solutions

[mangastudent]

If you're a prosumer like myself, who's been personally using magnetic tape since the late 1970s, you don't buy the very latest generation of LTO, you go back one or more generations. If you're good at scrounging, you should be able to get a tape drive with a lot of life for $1,000 or less, as I did in 2011 for a new HP LTO-4 drive which I'm still using, when LTO-5 was the new hotness. Today, without going to any real effort, I could buy a new LTO-5 drive from Newegg for $1,700.

A SAS controller is pretty cheap, ones based on the LSI200x chip particularly so right now. If you're patient, you only need a buffer device to avoid shoe-shining your tapes, rather than a fast RAID array. That is, you have your backup software write a 10GiB or so file at a time (probably ought to be larger for the newer LTO generations), and then stream it to the tape drive in one go. I use a Cheetah 15K drive right now, a DRAM tmp file would suffice, you can also scrounge higher end used but not used up Intel DC for enterprise datacenter 2.5 inch drives that were presumably taken out of service because their "small" capacity could no longer be justified.

All in all its quite practical, you have to do the math, including safe deposit box rental or whatever you choose for your offsite storage, and see if it makes sense compared to cloud vendors. Or you might not trust them to exclusively do your offsite backups, tape gives you a level of control you don't get from them.

Re:long term solutions

[lgw]

3-9s? That's worse than using 10-year-old HDDs, tossing them in a box, and then losing the box. I think you're off somewhere, but then is is MS so who knows. They sell their cloud services at a deep loss, so eventually that will change.

The best cloud archiving is S3 Glacier. It's roughly $50 per TB per year, and gives 11-9s SLA. You can avoid bandwidth costs with Snowball (or if you think in EB instead of TB, Snowmobile).

LTO8 is roughly $10 per TB, plus whatever Iron Mountain charges to store a box full of them for you, probably less than $1 pre TB per year, unless you're small scale. It's also an easy sell to auditors, where S3 may take some convincing.

Re:long term solutions

[Anonymous Coward]

Also, LTO-8 has only 12 TB of native capacity. Ostensibly, you can get up to 30 TB of storage per tape, but that's the best-case scenario. Realistically, you need to assume that you'll get 12 TB per tape, and if you get more, yay.

That’s around the density of hard drives.

More to the GPs point, a single drive is nothing, you have to look at disk library + disks + maintenance and what upgrades are possible VS tape library + tape drive(s) + tapes + maintenance, possible drive/jukebox upgrades etc.

For what it’s worth, you can upgrade a $4k tape drive a whole lot quicker than $4k of HDD.

Re: long term solutions

[Anonymous Coward]

Ive never had an issue getting the compressed data amounts onto lto 5 tapes. I dont say any reason that lto8 would suddenly be different.

That said im looking at this from a large scale commercial archival perspective. And in those cases the tapes go off site to an environmentally controlled store.

Then you look at the ease of use you get when you add in tape libraries. For example ones that hold 24 tapes you start to see why they beat other storages.

It doesn’t take much, photos and videos won’t really compress, but yah mostly compression is not a problem.

Compression ratio is also directly related to throughout (sorta obvious), but most people I’ve met don’t know that increasing block size will increase your average compression ratio too, because the drive is doing it, bigger write block is a bigger block for the compression algo to work it, I’ve seen it improve all they way up to 512k then sort of flatline. Please don’t use the default 64k :(((

Re:long term solutions

[lgw]

There's another factor you didn't consider that limits tape usage to very high volume use only.

Tape is rarely cost effective for personal backup (though I'm sure there's a Slashdotter out there with a 1000 TB hentai collection), but for any company that has a meaningful amount of data it makes sense for archiving.

You need to be thinking in terms of multiple hundreds of terabytes to justify the initial equipment costs, even if the tapes themselves are affordable.

Yes, though calling "hundreds of terabytes" a lot of data sounds odd to me.

Re:long term solutions

[lgw]

Removable disk hard drives have been here since the earliest days. Sealed disk was an invention along the way. It isn't an accident that the latter technology now dominates - you just can't do modern HDD densities without a controlled environment. Heck, all my recent HDDs are helium-filled.

Tape gives you a much larger surface area for a removable cartridge than you're ever going to get with disk, which is why latest-gen tapes generally store more than latest-gen HDDs, despite the limitations of "uncontrolled air".

Of course, SSD is a different world. Very dense, very pluggable. Still very expensive, though, and no promises at all about retaining data when unpowered. I suspect in a few decades SSD will finally replace tape, but the industry is far from that point today.

Re:long term solutions

[Anonymous Coward]

$150ish for 30tb of storage using LTO 8. WD Red 10tb drives are ~400ish. So storage cost is more like 8 to 10 times cheaper. And thats enterprise grade vs "prosumer" grade.

On top of that you need 3 drives to equal 1 tape. So storage costs are 3 times higher. Also I sure as hell wouldn't want to rely on a HDD spinning up for the first time after sitting on a shelf for 10 years, let alone 20 or 30 years. Sure it might, but I wouldn't want to rely on it.

As far as I am aware tape is currently the only viable long term archival storage method.

hieroglyphics can still be read today, thousands of years after being made. beat that for longevity.

Re:long term solutions

[Anonymous Coward]

You mean:-

https://en.wikipedia.org/wiki/Disk_pack

Re:long term solutions

[Anonymous Coward]

What's more, you've described an 8K black and white movie. 10 or 12 bits per channel would be more customary.
I don't know what studios do though. First thing would be to use 4:2:2 instead of 4:4:4. It's still too damn big? I think most people would want to use some lightweight compression, otherwise 8K looks dead to me except for Olympics or quarter billion dollar movies.

Re:long term solutions

[Anonymous Coward]

As is nearly always the case, there is a 'right' technology for your use case.

Tape (such as the LT0 8 mentioned before is a good choice if cost is a factor and particularly if you want to retain data for a very long time. I'm pretty certain I've read that LT0 8 tapes are certified (maybe expected) to last 30 years. If you're archiving very large quantities of data and will rarely need to access then tape is probably the right approach, particularly if you're using appropriate storage facilities.

Disk is a good choice when you need your data to be online. Plenty of places need this. Goggle is a good example of a massive data set that has to be accessed immediately and they can never be certain if a piece of data from today or 20 years ago is going to be needed.

Disk pacs could be a good alternative but they do seem to have disappeared. Older disk pacs (like the one from the link) were only reliable to a point. You would want to be careful using them for long term storage due to the relatively high risk of physical disk damage. This did not happen often but it happened more often than you would like. Whether a modern version could be produced that gave the cost effectiveness of tape I can't say.

Re:long term solutions

[Anonymous Coward]

At $150 for 8 TB, that comes to 640 TB. Realistically, that ought to be enough for anybody. :-)

Obviously you're not familiar with my porn collection

Re: long term solutions

[Anonymous Coward]

I think they strive for compatibility two generations back

Re:long term solutions

[Anonymous Coward]

As it turns out, the cost of the drives, even an array of autoloaders, is negligible when compared to the alternatives.

Every time I hear about some institution getting hit by ransomware or something similar, I have the same thought: "Damn, they could have put one hell of a robust backup solution in place for less than the $30,000 ransom they just paid there." The cost of losing all of your data in a fire/hurricane/flood/mudslide/meteor shower/whatever is orders of magnitude greater.

It's one thing to buy these units for personal use, but amortizing them over X number of years has been standard practice in the enterprise and even in midsized businesses since about forever.

Re:long term solutions

[Anonymous Coward]

What about backup to EPROM :)

Re:long term solutions

[mangastudent]

For example Microsoft charges $720/year for 30TB off archive storage, plus a few hundred for bandwidth. I doubt you could do it cheaper with tape, especially when you factor in periodic upgrades, duplication and operating costs.

I think that'll entirely depend on the quantity of data you're storing, and how much you need to access it once it's stored. In terms of raw costs, Azure Archive level blob storage gets down to $0.02/GiB/month for one datacenter no geographic redundancy, whereas as a prosumer I can buy LTO-4 through -7 tapes in quantity 20 for less than that at GB levels (haven't tried converting to GiB).

So I've got a budget of ~$0.22/GiB/year for the overheads you mention, and that's before I have to make any accounting for getting my data back, which at the base price costs $0.087/GiB, plus for Archival storage there's some small retrieval costs as I recall. That cost is fine if I've recovering from a physical disaster while making a 5-6 figure insurance claim, but lots of organizations need more routine, non "archival" access to their datasets.

Re:long term solutions

[AmiMoJo]

3-9s is under 9 hours of downtime a year. It's hardly terrible, especially compared to what many companies use for in-house DIY solutions.

As for S3 Glacier it is cheap, but I think you are misunderstanding the 11-9s number. It's not an SLA for general availability of your data, in fact it's not even an agreement at all. It's their average reliability per year, in other words a 0.000000001% chance (1 in 1 billion) chance of losing your "archive", which is a zip or tar file up to 40TB. It's a marketing claim, there is no compensation if your data is lost in the service agreement.

Of course you might have to wait while their system is down for maintenance to actually get your data. They don't seem to publish an SLA for the service.

Re:long term solutions

[Chaset]

And several "SyQuest" drives before that. Those 44MB/88MB cartridges were industry standard back in the day.

I even had a SyQuest EZ135, which was an attempt to compete with Zip. It had better performance than the Zip, but there was no way they could make those platters cheaper than a Zip disk.

One problem was the reliability/durability implications of people treating spinning platters the same way they treat tape. The other was the inability to keep up with the densities possible in fixed-spindle drives when they have to account for the mechanical tolerances of removability.

Re: long term solutions

[mangastudent]

I think [the LTO consortium members] strive for compatibility two generations back

Yep, normal rule is write one generation back, read two generations back. This only failed during their switch from metal particulate (MP) to barium ferrite (BaFe) as magnetic media, LTO-8 tapes won't read LTO-6, where either MP or BaFe magnetic media was allowed.

Re:long term solutions

[Waffle Iron]

This claim of a one in one billion chance of losing your data is also patently false.

The annual odds of a large comet hitting the earth and wiping out the human race is higher than that. Without any humans, there's no way to retrieve the data.

Re:long term solutions

[Marxist Hacker 42]

When Intel was searching for the plans for the 486, they found they had to bake tapes at 120F for an hour so that they could be read ONCE in a single pass into an image to recover the data from. And that is just from backup tapes stored for 18 years.

I'd hate to try to read a tape 30 years old.

Re:long term solutions

[lgw]

Ahh, I see. But availability isn't very interesting for archives, where you generally have a few days to retrieve data (since traditionally that means having Iron Mountain deliver a box to you).

S3's SLA for availabilty is 3-9s IIRC. I think most AWS services are, though they've been working on doing better.

Really, though, cloud archiving will come down to "do you trust this company for the next 20 years", more than other considerations. MS and AWS at least have a track record for staying around and keeping services up, not sure anyone else is even worth looking at (especially not "what service will they abandon today?" Google).

Re:long term solutions

[mangastudent]

Sounds like sticky-shed syndrome. Maybe Intel was unlucky in the tape manufacturers they bought from? Or like way too many other companies, were careless about the environmental conditions they were kept?

But you raise a good point, we really don't know if these tapes will still be readable in 30 years. It's just that tape has about the best record in affordable digital longevity. If you want better and also proven, microfilm and fiche might be your best bet, assuming you can still buy it and the equipment for it. Or scrounge old pre-Winchester hard drives, or maybe write to acid free paper, I think you can get 1MiB or more per side that way.

Re:long term solutions

[dgatwood]

No, I was approaching it from the perspective of a small business that didn't want to back up to the cloud. Most businesses don't need data to be available after a decade. The probability of something still having any value after it gets deleted rapidly approaches zero after a couple of months, and if it existed within the last couple of months, it should be in at least a couple of full backups and possibly incremental backups.

Re:long term solutions

[dgatwood]

Who might need 640 TB or more of storage? Let me see.

Hear that whooshing sound?

Re:long term solutions

[drinkypoo]

What I mean is, in this case, that nowhere says a hard disk must come with its own spinning engine.

It's a cost and space issue. Making a good interface between the motor and the spindle which will function at 7200+ RPM is extremely nontrivial. It will take up space and it will add complexity to the system. Currently everything is mounted firmly to the same chassis, it's simple and convenient.

Re:long term solutions

[turbidostato]

"The "spinless hard disks" or SSD drives are much, much more expensive per Gigabyte than hard drives or tape."

Think about why there's such a big difference between SSD's and magnetic-anything and come back with your own answers.

By the way, for long-term/cold storage, won't you think a "drive" the size, say, of a shoe box could be a thing?

Re:long term solutions

[turbidostato]

"It's a cost and space issue."

No, it isn't. Look at the main article, we are talking here about entire buildings for long term storage. It is about, as I said, mainstream vs niche.

"Making a good interface between the motor and the spindle which will function at 7200+ RPM is extremely nontrivial"

Of course it isn't. And you need it that way why...? Maybe because you want mean seek times in the miliseconds league? And why hard disks are about A5 size or less? Maybe because we are talking about all-purpose hot storage?

Remember the old days? hard disks were open-air and about size of a fridge. Why there're no hard disks the size of a fridge and about seconds' seek-time anymore? Maybe because, just like tapes, then you would be talking about niche and then rising accordingly per-unit costs due to sunk investments?

Re:long term solutions

[Anonymous Coward]

Also I sure as hell wouldn't want to rely on a HDD spinning up for the first time after sitting on a shelf for 10 years, let alone 20 or 30 years. Sure it might, but I wouldn't want to rely on it.

What is the degradation mechanism on the moving parts of an enclosed hard drive that is not powered? Why wouldn't it work?

Re:long term solutions

[mangastudent]

What is the degradation mechanism on the moving parts of an enclosed hard drive that is not powered? Why wouldn't it work [after 10 years on the shelf]?

Lubricants aren't necessarily designed to survive that long, especially at room temperature. Glues can eventually fail. Heads parked on disk surfaces might build up too much stiction. The plastic in the flexible ribbon cable that connects the fixed electronics to the disk arm might degrade.

Then there's non-moving stuff, flash memory will lose its charge given long enough. Other electronic components might fail, although that's probably a tiny thing, modulo what I don't know about the effects of higher temperatures needed for lead free soldier. The move to that was very badly managed, tin whiskers for example could make bridges.

Re:long term solutions

[Anonymous Coward]

Remember that platters need to be particularly safe from dirt and dust, which is why they're completely enclosed.

For exchangeable platters, there's

Solution 1: You enclose the heads with hte platters. Not a big savings to be made because you only factor out the motor but gain mechanical couplings and an electric connector.

Solution 2: You don't enclose the head with the platters. Now you need an opening for hte heads on the platter cartridge, a way to calibrate the head to the tracks on different platters, and a shutter that reliably keeps the dust out. Which is never perfect, so recording density and reliability go down.

Neither solution seems viable to me, but maybe someone can enlighten me how LTO-8 deals with dust.

OTOH, there's the risk of a dying tape drive. When that happens, you're practically forced to buy anohter one or you lose access to your tape library. I'd rather suffer the occasional death of a HD.

Re:long term solutions

[Xenx]

dgatwood replied and explained what he was talking about. I was just simplifying my response. My point was that they were explicitly talking about being the ones doing the local storage. If you're specifically trying to do the local storage, cloud storage from another provider will never be the answer.

Re:long term solutions

[Antique Geekmeister]

They're already quite common, they're a major part of the market for large, USB connected, spinning drives. They can require a bit of thought and caution to avoid disconnecting them while data is being written to the drive, but they're quite effective backup systems in small environments that cannot afford and do not need tape backup.

Re:long term solutions

[Marxist Hacker 42]

Or for that matter, archive to laser etched granite. But even that you have to keep indoors (at the very least).

Re:long term solutions

[Anonymous Coward]

I think the "9"s are about bit loss. If you have a 1 in 1 billion chance for bit-rot and a terabyte of data, that's almost 9000 bits of corruption.

Never underestimate the bandwidth of a..

[pipedwho]

..truck loaded full of magnetic tapes.

Re: Never underestimate the bandwidth of a..

[Type44Q]

It was mildly funny twenty years ago when it was a station wagon fully of floppies.

The first time.

Twenty years ago.

Re: Never underestimate the bandwidth of a..

[pipedwho]

Well, it was "a station wagon full of tapes hurtling down the highway" when I first read it Tannenbaum's Computer Systems textbook. And it's still funny twenty years later (mildly or otherwise), because it's still true.

In fact the SETI project still does exactly this to deal with insufficient internet bandwidth by transferring data in the form of a big pile of magnetic tapes from the radio telescope array in Puerto Rico to California for analysis. Which as alluded in the summary is a problem that may exist with the amount of data generated by this new telescopic array. And the reason I made the initial comment.

AI compression.

[Ostracus]

Construction is slated to begin as soon as next year on the Square Kilometer Array, a radio telescope with thousands of antennas in South Africa and Australia meant to detect signals emitted more than 13 billion years ago. It's been estimated the project could generate an exabyte (1 billion gigabytes) of raw data every day, the equivalent of 300 times the material in the U.S. Library of Congress and a huge storage headache all by itself

Good thing we just had a Slashdot article about intelligent compression. Even though most poo pooed it as not needed.

Re:AI compression.

[Anonymous Coward]

For a radio telescope survey, it might very well not be needed.
The AI intelligent compression article reminded me of this:
https://en.wikipedia.org/wiki/No_free_lunch_theorem

Re:AI compression.

[jythie]

Heh. Compression on scientific data is a nightmare. I would not trust AI to do it... I wouldn't even trust grad students to do it.

about 10 years ago

[Anonymous Coward]

I was talking with a coworker still at a former employer. He said their tape budget was up to $100k/month, that they were doing nearly a petabyte every night in backups. This company is still around and grown bigger since. They're not as big as Google, but they're up there.

Does not surprise me that this is still big business.

Re:about 10 years ago

[Anonymous Coward]

It used to trade as OMTR.

FINALLY. Something I can understand.

[grep -v '.*' *]

the project could generate an exabyte (1 billion gigabytes) of raw data every day, the equivalent of 300 LoCs.

Bytes, meters, inches, gallons, tons, carets, troy ounces, femtoseconds, microwatts -- pshaw, I was WONDERING when we were going to get back to normal units of measurement. Now who wants a pony?

Back on Topic: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. -- A. S. Tanenbaum

Re: FINALLY. Something I can understand.

[jd]

Unless it's in a convoy, the bandwidth is low. The latency is terrible.

Re:FINALLY. Something I can understand.

[Anonymous Coward]

think of the LOC as a class/object library approaching IDE proportions, the facets of measurement are limited only by the scope of your feable imagination

Does magneto-optical tape exist?

[Miamicanes]

Is there any particular reason why nobody makes a product that's basically like non-LTH (phase-change/magneto-optical) BD-R, but on a flexible film substrate stored on reels instead of bulky discs?

The main problem I see with magnetic tape is that it's inherently susceptible to stray magnetic fields (including the Earth's poles). In contrast, phase-change media can theoretically have a passive lifespan that's measured in decades (centuries, if "being able to read it with normal, consumer-grade hardware as a normal OS filesystem" isn't a hard requirement, and you can deal with forensic data-recovery using exotic purpose-built hardware).

Re:Does magneto-optical tape exist?

[Anonymous Coward]

How about 150dpi laser printed QR codes on 8.5x11 sheets of archival grade paper?
Very low density, excellent lifespan. Readable with any desktop scanner!

Re:Does magneto-optical tape exist?

[Anonymous Coward]

Same reason why nobody makes airplanes from cheese. You don't get working technology by combining random sh*t.

> In contrast, phase-change media can theoretically have a passive lifespan that's measured in decades

Tape's life time IS measured in decades.

Re:Does magneto-optical tape exist?

[Anonymous Coward]

Maybe finding a drive to read your media with will be the overarching problem in a half century or less?
Not insurmountable but exotic lab hardware sounds very slow, like reading a petabyte at bytes per second rendering the idea practically useless. Of course we're talking centuries so anything might goes, but by then you're effectively building a tape drive from scratch.

I have another question : can we have a new super floppy again? Maybe it would be technically doable to build something of a silly high capacity like a 1TB floppy - setting aside whether it's magneto optical or something different. But let's imagine a 10GB floppy or some such. (small, sturdy, can have a dual drive on USB 3.0 and computer doesn't grind to a halt when using it).

The main draw is to go back to plugging media into a computer, not a peripheral which itself is a whole computer with all the nasties that can happen. In fact you can now get > 1GB/s flash drives, on Thunderbolt : so instead of plugging a USB or perhaps external SATA device we're now supposed to plug a PCIe device into a computer to get data. A fraudulent drive might do anything (how about : presenting itself as a screen or goddamn video card to your computer, tricking it to mirror your screen and grabbing shots at will)

Re:Does magneto-optical tape exist?

[hankwang]

I'd guess that it's hard to optically read tracks that are spaced at a micrometer apart from each other when all 10000 tracks across the width of the tape need to be read in parallel. If you look at the read/writd head in a dvd player, you see that it's about a centimeter for the lens and the suspension around it, and it can only read one track at a time.

You'd also need a tape substrate that is opticaly clear, thin, and still strong and capable of preventing the data carrying chemicals from degradation.

Re: Does magneto-optical tape exist?

[jd]

There was research into liquid crystal memory, which could have been used like optical tape, but it proved to be impractical in reality.

Re:Does magneto-optical tape exist?

[AmiMoJo]

The cost of developing such technology doesn't justify it. Tape is good enough, companies are used to storing it securely for the long term, and they already have tape drives to read/write it.

Re:Does magneto-optical tape exist?

[jabuzz]

These tapes are in massive libraries. Typically you string ~16 rack sized chassis together, which using LTO8 will give you around ~300PB of uncompressed storage. We are talking the likes of the IBM TS3500 or the newer TS4500. The other two players in the market are SpectraLogic and Oracle/StorTek who have similar libraries. With the TS3500 you used to be able at least to get a passover option so you could string 15 rows of libraries together for insane amounts of storage.

The only thing I doubt in the article is the rubbish about Iron Mountain. If you are Google/Facebook/Amazon etc. you just replicate your data to one of your many remote data centers. No point messing about having humans physically handling tapes on a daily basis (well other than feeding Audrey with new tapes). To be honest actually having your tape library onsite is a fairly dumb tactic anyway as there is a good chance the reason you need to use your backup is because the data centre has been transformed into a pile of smoking rubble or a large swimming pool.

Finally when you need to change tape technologies you just have the software copy it from the old tapes to the new tapes, and if they are all in the library while it might take several months it does not involve human interaction. Other than take old tapes out and putting new ones in every few days. You are going to need to do this probably every 6-7 years if you stretch it so the tapes only need to be able to reliably store data for say 10 years. That said provided it is stored in the correct environment LTO is good for 30 years from memory.

Re:Does magneto-optical tape exist?

[NeoMorphy]

As tape storage capacity increases you normally consolidate the older tapes onto the newer tapes to save on space and to eliminate to issue of not being able to read the tapes with onsite tape drives. We use automated tape libraries to handle the mounts/copies/ejects. Software knows whats on every ocr labeled tape and scheduling/tracking/expiring of tapes going to/from offsite facility is automated.

This has been standard procedure for decades. Nobody plans on tapes sitting out there for decades, they're constantly refreshed as technology improves to reduce the number of tapes and storage cost and to keep them compatible with current tape drives.

Re:Does magneto-optical tape exist?

[Anonymous Coward]

The only thing I doubt in the article is the rubbish about Iron Mountain. If you are Google/Facebook/Amazon etc. you just replicate your data to one of your many remote data centers. No point messing about having humans physically handling tapes on a daily basis (well other than feeding Audrey with new tapes). To be honest actually having your tape library onsite is a fairly dumb tactic anyway as there is a good chance the reason you need to use your backup is because the data centre has been transformed into a pile of smoking rubble or a large swimming pool.

Best practice for a while has been disk-to-disk-tape. Trying to go from the backed-up client directly to tape will cause shoe shining for a while now (even with multiplexing streams).

So back up your clients to a VTL, replicate the first copy of the data to another DC to a second VTL, and then if you need to keep the data around for more than "x" weeks, replicate it a third time to actual-tape.

Generally one would also probably have snapshots on traditional file stores, so one probably doesn't even have to go to the backup software for recovering easy things.

Re:Does magneto-optical tape exist?

[BrendaEM]

Oddly, either that or DVD/CD tape was used in the 1980s move Brainstorm. It looked really cool, if you think about the density.
https://reellibrarians.files.wordpress.com/2012/04/brainstorm-closeup.jpg

Re:Does magneto-optical tape exist?

[mangastudent]

I'd guess that it's hard to optically read tracks that are spaced at a micrometer apart from each other when all 10000 tracks across the width of the tape need to be read in parallel.

That's not quite how LTO tape drives work, they have tape heads with 8 parallel read/write heads for LTO-1 and -2, 16 for -3 through -6, and 32 for -7 and -8. After writing a "wrap", a single end to end pass, the tape head moves a bit and writes another, partially overlapping the previous set of tracks in the style of SMR hard disks, this allows more density at the cost of requiring a total rewrite if you want to start over.

But your general points stand, except the tape substrate wouldn't have to be clear, you would layer over it a reflective layer, a writeable layer, and then one or more protective layers, with probably one or more buffer layers in-between. But writing at high speeds is a must in this domain, so some sort of multi-track optical head would be required, which strikes me as difficult. Especially fitting it into a 5.25 inch form factor

Looking at Fujifilm's brochures, I see that their metal particle (MP) generations of tapes uses 2 layers over the base, the magnetic material on top, and one between it and the substrate. For barium ferrite (BaFe) two additional layers are added directly to the even more thin tape substrate. And since this is a much more batched process, assume manufacturing tape reels at least 1 foot in diameter, vs. the sort of one at a time method for optical discs, which as far as I know includes a vacuum sputtering step for the reflective metal layer, plus a much thicker substrate, we can imagine why they're much more expensive per GB.

Re:Does magneto-optical tape exist?

[Anonymous Coward]

disk to disk replication is fine but what about retention? When there is a corruption it is replicated to the target disk and then you have the unknown time frame of how long it will be until the corrupted file is identified as needing to be recovered. It's one thing when it's facebook photos. Quite another when it's images of the official land ownership records for a state.

Re: Does magneto-optical tape exist?

[jabuzz]

Oh dear, someone else who's knowledge is well out of date. Since LTO4 the tape speed has been variable to avoid shoe shinning. That is the drive will adjust the tape speed to match the incomming data rate. That said I have only used TSM (all other backup products being rubbish in comparison) and yes best practice is to have a fast disk pool for you daily churn, which is then punched out to a primary and copy pool of tape. You only need a VTL if your backup software is junk (aka not TSM/Spectrum Protect) or you want a dev enviroment for testing ourposes and a real tape library is too expensive.

Re: Does magneto-optical tape exist?

[mangastudent]

Oh dear, someone else who's knowledge is well out of date. Since LTO4 the tape speed has been variable to avoid shoe shinning.

Which I can most certainly hear as my HP lower end LTO-4 drive reads data off my 15K Cheetah hard drive buffer. But what can the drive do if it doesn't get data for some number of seconds? How low is the lowest rate it will accept data without having to periodically stop the transport?

Re:Does magneto-optical tape exist?

[epine]

IBM TS4500 R3 Tape Library Guide

Increased capacity: The TS4500 can grow from a single L frame up to an additional 17 expansion frames with a capacity of over 23,000 cartridges.

IBM TS1150 tape drives deliver the fastest and largest capacity drive for enterprise archiving and data protection

Uncompressed cartridge formatting of up to 10 TB with the use of IBM Tape Cartridge 3592 Advanced Data (Type D).

Woof! The rest of the text would make paint cry.

That makes 230 PB of uncompressed storage.

But there's a more recent TS1170 with 40 TB raw capacity. So we could be looking at a raw 0.92 EB.

But that's assuming the new drive fits into the old frame, and I haven't sacrificed any recently warm chickens to the product matrix oracle on this quick scrape, so who's to say?

Re:Does magneto-optical tape exist?

[Miamicanes]

That would be silly... but what if you took the basic idea and applied it to laser-exposed microfilm? Sure, it might be slow to expose & probably require off-site developing... but once you had it done, you'd have a storage medium that could passively survive decades, maybe centuries.

The fundamental problem with any data-storage mechanism that requires active handling every few years to preserve it going forward is the fact that such data is unlikely to survive long enough to BE of interest to future historians. Businesses pay to retain records long enough to satisfy regulatory and immediate business needs, and that's IT. Governments are only slightly better at long-term storage for posterity. Just look at 1970s-era NASA and the BBC.

Old data that HAS survived to the present time almost ALWAYS survived ONLY because it was in a form that could be put somewhere, forgotten about, and rediscovered decades after it would have otherwise been systematically destroyed... and often, only because it WAS someplace that the people who would have wanted to see it get destroyed either couldn't get at it, or didn't realize it even existed. Like old letters in an old chest that went from attic to attic for a century or more. Or old tax records from Edwardian England or the Roman Empire. Or crates of punch cards from some long-gone business that were stored in an old office building in Detroit that was abandoned and left to rot decades ago.

Consider for a moment the fate of really, really old films from the early 20th century. Most of them were recorded on nitrate film that was literally dangerous. Few of them were considered valuable enough at the time to be worth transferring to "safety" film. The copies that exist today exist primarily because of a few film-buffs who were willing to disregard safety, grabbed them from trash heaps, and stored them in backyard sheds, until their grandkids discovered them decades later while cleaning out the house after their death & realized that they were a) dangerous hazmat that could literally go up in flames at any moment, and b) now worth a fortune, because all known surviving copies of them had been (intentionally or unintentionally) destroyed decades ago.

Museums and universities routinely have to deal with archived data that they don't have the resources to deal with RIGHT NOW, but nevertheless would LOVE to have some way to put "on ice" and just passively & compactly store somewhere indefinitely on the off-chance that some future historian might find it interesting. Or until AI gets to the point where historians can use computers to analyze, catalog, and categorize data that would take a hundred human lifetimes to meaningfully sort and categorize by conventional means. At the time the decision is made to archive it, the data might be very low-value, and won't BE sufficiently valuable to expend resources re-archiving for several human lifetimes. But if it's stored on a medium that can literally be packed into a mountain and forgotten about for a century or two, that's ok... it'll still be there for future historians when they eventually DO care about it.

Re:FINALLY. Something I can understand.

[Anonymous Coward]

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. -- A. S. Tanenbaum

That's such a tweed-jacket thing to say.

write bandwidth...

[johnjones]

transfer bandwidth is something quite different from job complete time

by the time you write to all those tapes the Square Kilometer Array will have generated more data...

they wont be using tapes for transfers...

Re:write bandwidth...

[pipedwho]

I was going for funny. But, theoretically assuming the antenna array was operating continuously 24/7, then a few thousand tape drives all writing in parallel would significantly reduce the write bandwidth problem.

The write bandwidth of any tape backup array needs to be at least as much as the average capture/save bandwidth of the data, otherwise it would never complete. In reality, that array would either pre-process some amount of data before archiving and/or only capture data in bursts - so total archival bandwidth my have spikes, but average to a readily achievable volume for archiving.

You Forgot

[Anonymous Coward]

Billion-dollar class arms purchases. Only then you are allowed to wield the bone saw.

"a ... bug deletes thousands of Gmail messages"

[Anonymous Coward]

...and nothing of value was lost.

Hey, c'mon: if you trust your valuable data to Tah Cloud, you deserve a hard rain.

So long as the sync is good

[jd]

Parallel file store onto striped tape systems provides the best throughput for serial data.

Scientific data can be stored using parallel NetCDF, which is designed for such cases.

Tape is useless for random access, but that's not what you do with backup/restore or simple data logging from scientific instruments.

Science breakthrough flip the switch

[Anonymous Coward]

I bet if Bloomberg exist in the old floppy day, their editor/journalist will argue that company should invest to improve the floppy disk capacity, flash memory will not pick up as it is "too expensive".

I am not doubt that a science breakthrough in storage technology emerge from somewhere will render magnetic tape extinct.

Durability

[Meneth]

What could ever replace the durabiliy of magnetic tape? Duct tape, maybe.

Isn't it funny?

[DMJC]

How obvious it is that a monopoly on magnetic tape is a bad thing, and yet the USA allow monopolies over all kinds of things involving last mile infrastructure and other critical services. But this one costs the tech companies so it's OK for the government to intervene but anywhere else and "IT'S TOO MUCH REGULATION".

Re:Isn't it funny?

[virtig01]

How is the government intervening here? The two competitors are lobbying the government to ban the other, and the government is refusing to intervene.

What part of the cost is tape cost?

[mapkinase]

>meaning higher costs for the big cloud providers

What part of the cost of the cloud-keeping is tape cost?

Cant you just

[Anonymous Coward]

You know backup your cloud to another cloud? Who needs this old fangled tape tech. LOL!

Re: "a ... bug deletes thousands of Gmail messages

[Anonymous Coward]

Yes, because we all use these services for non-valuable data. That's what makes these services sos valuable, d'oh.

Tape drives

[Daralantan]

This just makes me remember at some point in the 90s when I was struggling with my 1GB hard drive and wanted more storage space. I'd heard about tape drives and DESPERATELY wanted one. I knew nothing more than they could hold a lot of stuff. Would be cool to just keep around as a memory these days.

I'm pretty sure I was under the impression that I could install computer games and such to it at the time, but I'm not sure if that's what I actually thought.

xkcd to the rescue

[Anonymous Coward]

https://xkcd.com/1737/

What comes around, goes around

[Anonymous Coward]

With backup many are more concerned about cost for capacity then speed or efficiency. I know many smaller businesses who still use tape drives for backup and do not plan on upgrading any time soon. Just as many people still find the capacity and cost of HDD far less then SSD's.

Re:Disks are superior

[110010001000]

The only advantage magnetic tape has is it can hold more data.

Classic Slashdot

Fahrenheit 451

[bugs2squash]

Get congress to act. If they can reduce the size of their library to, say,500Mbytes, then 300 LOCs will fit on a thumb drive. Problem solved.

LHC

[Anonymous Coward]

That's 25 Gb/sec after throwing away most of it. The sensors do the first pass, as the wires doesn't have the bandwidth to get all of the data out of the sensors.

Re:Disks are superior

[mangastudent]

It's completely unclear the BD-R optical discs consumers can get their hands on are superior, let alone more durable. CMC seems to currently be the least worst manufacturer, all the higher quality manufacturers have stopped making them, and they're obviously not highly trusted because of their history with previous generations of optical media.

For DVDs, I'd go with MAM-A, silver or gold, ditto CD-Rs, which I trust a lot more than DVD recordable media, since pressed DVDs pushed red laser CD technology as far as possible. (Taiyo Yuden exited the optical disc market in 2015, selling their stuff to CMC.) Therefore not going to calculate their costs, especially since their small capacity will start to really affect your off-site storage costs, unless you can stash them with friends or family, and trust them to keep the environment in which they're stored within the requirements (both tape and optical discs are picky here, that's the one advantage hard disks have over them.

Now we get to capacities, if you're going for low costs, single layer is where it's at, 25GB for BD-R, 4.7GB for DVD-R. Compare to 800GB native for LTO-4 tape, back when they were not ancient you could get new high quality Fujifilm ones for ~$22 in lots of 20, I now see a price of $14.70. I see today that Newegg is selling LTO-5 1.5TB native quantity 1 at $23, LTO-6 2.5TB native at $32, and LTO-7 6TB native at $82 (that's less than $1/TB more expensive than LTO-6), and a quick check at Amazon shows their LTO-6 and -7 prices are not competitive, even before we get into quantity discounts, which are the standard way to buy tape.

Comparing my first purchase of 25 Verbatim 25GB BD-Rs just this month from Amazon, to a quantity 20 price from a 3rd party I trust, Malelo and Company, for LTO-6 tape, we're talking $0.0352/GB vs. $0.0105/GB. LTO-5 weights in at $0.013/GB and LTO-4 at $0.0184. And I trust tape from Fujifilm infinitely more than I trust BD-Rs from CMC. Ah, Verbatim at quantity 50 BD-Rs only gets you down to $0.0306/GB.

Hard drives

[Anonymous Coward]

I like hard drives, but one of the advantages of tape is they ship well.

Sending a box of tapes Fed-X off site is safer than a box of hard drives.

Drives are by nature more fragile.

Re:So why doesn't somebody GNAA GAY NIGGER FELCH

[Anonymous Coward]

You must be new here. GNAA has been around /. for over a decade...

THESE guys still make magnetic tape

[p51d007]

https://www.nationalaudiocompa... Business is BOOMING. Not sure if they are into the mag tape storage/computer stuff, but boy do they churn out cassette & reel to reel tape.

MOAR and MOAR features

[DarkRookie2]

Even though we haven't found anything better than tape yet.

Re:Disks are superior

[Anonymous Coward]

A few years back, I upgraded my storage server, new CPU, RAM, and a half dozen 4TB drives in a raidz2. Went from a few hundred gigs on failing old drives to 15 TBs. I had more space than I knew what to do with, so I started trying to bring everything I had burned to CD and DVD back online.

My old CDs from the 90s worked ok (yay Verbatim and old school TDK!) and I was able to read almost all of them after some cleanings. The DVD-R's were a different matter entirely. After as little as 5 years, they had degraded to the point where they were about 70% readable. No amount of cleaning or swapping out drives made any difference. Fortunately, this was mainly downloaded movies and music that could usually be easily replaced, so no big loss. As it turns out, consolidation of that market led to a race to the bottom in terms of ink quality and all of our burned DVD-Rs and BluRays are doomed to fail, unless we paid for pricey "archival quality" ones. I didn't.

Contrast this to work. At a former job, I recall digging a long retired LTO drive (an original one) out of storage in the basement and sticking a tape that had sat on a shelf for 10 years in it, and being able to pull files with no issues at all. Those things just work.

Re:Disks are superior

[mangastudent]

DVD-R technology is very marginal, the original pressed DVD version pushed red laser technology as far as it could go. But you still have to buy quality media, for CD-Rs I went with Taiyo Yuden (early on branded as Fujifilm in the US) and not a single one has failed me yet.

For DVD-R I got some from Taiyo Yuden as well, but didn't trust them if for no other reason than that they only cost a cent more than their CD-Rs. MAM-A gold DVD+Rs were my target for "archival quality", ought to test the few that I cut, but they were exposed to bad environmental conditions for a week or more so if they fail that won't tell me anything (note that Taiyo Yuden exited the optical disc business in 2015, selling their stuff to CMC).

Per my readings this month, comsumer BD-Rs are now a disaster, with CMC making the least worst (!!!). No joy there, and as you say, tape just works, still using the HP LTO-4 tape drive I bought in 2011 for ~$1,000 (it doesn't get a fraction of the wear a business would like put it to). Bought 60 tapes of Fujifilm and HP in quantity 2012, 2013 and 2014, plus some odd lots of those brands, and they're still doing fine.

Spam?

[Anonymous Coward]

So the post says that this is a Bloomberg article, but Arstechnica is linked. Also, when you click one the link ", magnetic tape lives on as the preferred medium for safely archiving critical cloud data", it brings you to a Bloomberg page saying that they've detected suspicious activity on your computer.

Holographic storage?

[kimgkimg]

Whatever happened to the holographic media that was suppose to kick tape to the curb?

Re:Holographic storage?

[Anonymous Coward]

Thank you for asking this!! I was thinking the same thing; I can't believe nobody else remembers this was a thing!!

Seriously, we should have the tech for better long term high density storage by now!!

Re:FINALLY. Something I can understand.

[Matheus]

A unit of measurement that requires calculus to translate. Since the amount of data in an LoC is continuously changing by a variable amount of delta... I mean is there an LoC service I can ping like NTP to get the current value of its scale? Can I get access to historical data? I'm trying to read an article from 1992 and am having trouble understanding the scale it's trying to convey in 1992 LoCs!

I really need to sort out the conversion algorithms to other useful units of measure like:
LHCDO (Large Hadron Collider Data Output)
GWC (Google's Web Cache)
Parsecs (Because why aren't we measuring storage in Parsecs?? If it's good enough for the Kessel Run it's good enough for storage needs!)

Tapes not reliable

[omfglearntoplay]

10 years ago we backed up using tapes. Always always always had issues with reliability of the data on them. Talking to peers, everyone experienced these problems. I don't see anybody talking about it here... am I missing something?

Re:Tapes not reliable

[mangastudent]

You're missing something. I've used tape since the summer of 1978, DECtape then, BASF magtape in the 1980s, Sony DDS in the 1990s, Fujifilm and "HP" LTO-4 in this decade, and never had problems. Perhaps you bought tape from one of the companies that just happened to drop out of the market? Sure you stored them properly, and didn't let the drive temp get too high while writing them?

This is the first I've heard of LTO tapes having systemic problems, aside from one machine room incident where the HVAC failed and it was at 120F or more for an extended period.

Re:Tapes not reliable

[BeaverCleaver]

Unless it was with audio tapes on a Commodore. Those were pretty flaky.

Re:Tapes not reliable

[mangastudent]

Yeah, that's not a use for which the tapes were intended. I still got the impression in the late 1970s from my friends with various audio tape systems attached to their microcomputers that the Commodore PET's was the best of all of them. AlphaMicro also had a VHS videotape based backup system around that time, which my family decided not to mess with. DECtape of that era? Rock solid in my summer's use of one. Magtape in the 1980s, ditto.

Magtape in the 1950s was used as an alternative to cards for primary storage for IBM's business computers until they solved the hard problems of hydrodynamic floating disk heads, they had to do some hard science and engineering using a drum based IBM 650 to crack that necessary problem. Oh, and business customers were much more demanding of reliability than scientific ones in those days, they basically rejected Williams tube based storage, demanding core memory.

Through at least Y2K magtape, maybe not always by then 1/2 inch reel to reel 6250 bpi, was used for primary storage of all sorts of big data sets. Now of course it's just used for backup and archival purposes, LTO tapes write partly overlapping tracks like SMR drives to get maximum density, you can't rewrite a particular tape block or file.

They're still very good at what they do, as long as you observe the environmental requirements (from memory for LTO-4, humidity less than or equal to 50% and "room temperature" for long term storage, you can let them get up to 60% humidity for up to 6 months), and I suppose avoiding buying garbage tapes, that might be a reason some of those 4 companies dropped out of the market. I certainly would never have bought any of their tapes, Fujifilm and Sony all the way.

Re:Seriously doubts

[mangastudent]

Tape drive prices are all over the place, there's lots of variables there including of course speed and generation, but you're completely off about media. Amazon prices from a good 3rd party company I've done business with before, Fujifilm tapes, for LTO-6, we're talking $0.0352/GB, LTO-5 weights in at $0.013/GB and LTO-4 at $0.0184.

Newegg prices for 8TB drives, the current sweet spot in capacity, range from $0.0256/GB to $0.0325 for the lower end of the 5 year warranty 550TB bandwidth/year enterprise drives. I hope I don't have to look up per GB prices for the higher capacity drives, companies by them and the higher capacity LTO tapes because space for them isn't cheap.

The LTO drives do cost 10x a hard drive, but you can put cheap media in them, and all day, your claim they are only meant to run a few hundred tapes before breaking, and the tapes are "even much less reliable" is falsified by my own experience with an HP LTO-4 drive and 65 Fujifilm and "HP" tapes over the last 6 years, incremental backups every day, full 3-5 tape backups once a month (and was doing Bacula differentials every week for some years).

Enterprises use them because they're very reliable, less fragile than hard drives which makes them good for off-site storage, etc. Cloud companies, which this article used as a hook since they're currently sexy? Only for archival storage, maybe, like AWS Glacier and Azure Archive, obviously any class of storage that's available within less than a second isn't going to be on tape. See also all the scientific uses where 100s of petabytes which don't have to be frequently accessed are reliably stored on LTO tape (you think they're willing to lose data and thus papers and their careers???).

Re:Seriously doubts

[mangastudent]

Correction, for LTO-6 tape we're talking $0.0105/GB, that $0.0352/GB price was for Verbatim single layer BD-R disks in quantity 25.

Time to get back to piano rolls

[Anonymous Coward]

https://www.google.com/search?q=piano+rolls&source=lnms&tbm=isch&sa=X

The original digital recording medium!