Facebook Puts 10,000 Blu-ray Discs In Low-Power Storage System

itwbennett writes "Facebook said last year that it was exploring Blu-ray for its data-center storage needs, and on Tuesday it showed a prototype system at the Open Compute Project summit meeting in San Jose, California. It designed the system to store data that hardly ever needs to be accessed, or for so-called 'cold storage' (think duplicates of users' photos and videos that it keeps for backup). The Blu-ray system reduces costs by 50% and energy use by 80% compared with its current cold-storage system, which uses hard disk drives, said Jay Parikh, Facebook's vice president of infrastructure engineering." It's a prototype, and they're also evaluating low power flash as another alternative to keeping seldom accessed data on hard drives.

Comments

Write once?

[WPIDalamar]

Anyone know if these burners are write-once drives?

If so, it pretty much guarantees that Facebook keeps a copy of your stuff forever, even if you "delete" it.

Re:Write once?

[tripleevenfall]

Doesn't Facebook have a right to control over their product? (you) ::ducks:::

Re:Write once?

[StripedCow]

When you delete your account, somebody will go and get the corresponding disk, copy it (except your data), and destroy the old disk.

It's write-once only if you don't consider "destroy" a write-operation.

Re:Write once?

[lagomorpha2]

If so, it pretty much guarantees that Facebook keeps a copy of your stuff forever, even if you "delete" it.

Facebook keeps a copy of your stuff forever, even if you "delete it". So does gmail/google. Even stuff you type into a textbox but never submit.

http://www.slate.com/articles/...

Come to think of it, "deleted data" is probably exactly what this cold storage is for. They never have to worry about overwriting it when users change the data because it's data the users have already "deleted".

Re:Write once?

[The Grim Reefer]

It's write-once only if you don't consider "destroy" a write-operation.

I'd say it's more of a delete-operation.

Re:Write once?

[mlts]

Some technologies come full circle. I'm reminded of Kodak's optical storage technologies that stored 3-6 gigs on large (8 inch?) MO platters in a jukebox that had its built in "clean room". The advantage of this technology was the fact that once burned, the data was there forever, which was useful for long term archiving.

Rewritable MO disks came out in drive arrays after that, arrays that had the ability to flip disks, so it could read/write the both sides (300 megs per side.)

Optical tech ended up on the sidelines because tape got cheap, especially when DLT started having decent capacities and tapes with WORM capabilities hit the data centers.

Now, tape drives are very expensive, and require a LOT of I/O on the attached computer, or else they will shoe-shine themselves into oblivion.

In the past, optical burners had issues, buffer underrun was one of those. Now, with modern ones that just turn off the laser once the buffer empties and resume very close to where it left off once data starts arriving again.

With tape out of the price range, I have not understood why someone hasn't made a Blu-Ray autochanger. Sony has one, but it is a carousel unit made for playing. However, couple that with a BDXL drive, "flippy" disks that have two sides for twice the writability, and that would provide more than adequate storage on an archival basis for large volumes of data. Two autochangers will allow one to have the ability to move data offsite, and almost every backup program out there has some form of encryption on it.

I just don't see why this isn't done. Even a 5-10 Blu-Ray autochanger that used five disk caddies (so one could just load the pack, and then not have to touch the media after that) would be immensely useful for critical backups.

Re:Write once?

[Chris Mattern]

When you delete your account, somebody will go and get the corresponding disk, copy it (except your data), and destroy the old disk.

Except they won't. Facebook doesn't delete your data when you delete you account *now*, what makes you think they'll do it when it becomes this much harder?

Re:Write once?

[Mysticalfruit]

I'd assume the drives are capable of using the many write media, it's then the economics of media cost. Also at some point is it easier to just discard the media then worry about a piece in a middle of a backup set going bad after X number writes?

I've just assumed all along that they kept everything. Deleted just meant "not seen".

I'd love to see the machine they built. A 10k stack of bluray discs at 1.2mm a piece is 12m. How are they storing the media so they can get to each piece? I'd imagine their using an ink jet printer to label each one. Maybe they're bundling them in stacks of 100 and labeling each stack?

So many questions!

Re:Write once?

[Grishnakh]

Huh? How would a 5-10 Blu-Ray autochanger be userul for backups? 10 BDs equals a paltry 360GB of data storage; that's only enough for 1/3 of a typical 1TB hard drive. BDs are nearly useless because they simply don't store much data; why FB is bothering with them, I have no idea. Optical discs have always been found to be pretty awful in terms of storage capacity and data integrity over time compared to tapes. The only problem tapes have is the drives are expensive, but large companies don't have a problem with spending $2k on a drive. On a per-GB basis, they're easily the cheapest thing out there.

Re:Write once?

[Anonymous Coward]

When you delete your account, somebody will go and get the corresponding disk, copy it (except your data), and destroy the old disk.

Of course they will.

BTW, I have some prime oceanfront property for sale here in Colorado, cheap. Interested?

Re:Write once?

[Guspaz]

How do you get 36GB per bluray disc? Commercially available BD-R discs come in a variety of capacities (25, 50, 100, 128) depending on number of layers and density, and 36GB isn't one of them.

10 BDs equals up to 1280GB of storage for quad-layer high density discs, although I can't find any of the 128GB discs for sale, only 100GB discs. Either way, the higher capacity discs are rather expensive, but if you're buying them in big enough bulk, it may not matter as much.

Re:Write once?

[Solandri]

There are larger capacity BDs, but they still don't amount to much storage. As others are speculating, this is probably a pre-emptive action by Facebook, so when they eventually get sued for not deleting someone's data, they can truthfully testify in court that it's logistically "too hard" to comply with a single user's delete request.

Re:Write once?

[nabsltd]

LTO6 is still pretty small at over 2TB per tape. No damn idea why Facebook wants to "stripe" that same dataset across 20+BDs.

T10000 tapes are the same size and store 5TB uncompressed, and cost no more than $300/each. That's about $0.06/GB, with Blu-Ray no less than $0.10/GB.

For storing less than 100TB, then hard drives plus infrastructure (drive bays, controllers, etc.) win on price/performance, and Blu-Ray or tape without auto-changer capabilities isn't far behind, while either with auto-changer is far more expensive. For storing thousands of terabytes, you're definitely going to need an auto-changer, so tape wins hands down.

Re:Write once?

[failedlogic]

There's a new (and rather expensive) Sony product: http://pro.sony.com/bbsc/ssr/m... that they're saying will be in densities up to 300 GB. From what I've read it looks like a Blu-Ray sized MiniDisc.

I'm not saying these are any better, but there is higher optical densities becoming available. They're targeting high-end video right now in the US markets. I tried to read-up more on this out of curiosity a while back but majority of info in on Sony.jp site site in Japanese. At $6,000 for the drive alone, hopefully there'll be some cheaper alternatives out there. The media prices are much less .. I see one site with a 300 GB disc for $50.

It seems Sony is partnering up with Panasonic. Hopefully something interesting happens with this.

I'd like a backup alternative to DVDs and doesn't seem to be worth the leap to go to Blu-Ray and most tape is too expensive for home use (and/or hard to find).

Re:Write once?

[Grishnakh]

Tape drives are a lot cheaper than $6k, and 2TB tapes aren't any more expensive than that $50 300GB disc. If the prices come down, then great, but if they're just targeting high-end customers, I don't see prices changing at all. Regular consumers don't have a need for any of this stuff. The only reason CD-ROMs and DVD-Rs made sense for a little while was that those formats were used by consumers for music and movies. Nowadays, everyone just uses MP3s/AACs on their hard drives and mobile phones for music, and streams video for movies from Netflix and Amazon.

Re:Write once?

[AmiMoJo]

BluRay discs designed for archival are actually thought to be very good for data storage. They are actually more like magneto-optical discs. A laser melts the plastic and a magnet aligns reflective particles within it, and then the plastic hardens again. It's pretty permanent if the disc is made of durable materials.

Re:Write once?

[kheldan]

I don't get why they'd use an opto-mechanical drive for this. Why not use SSD's instead? Yes I know TFA said they're looking into using "low power flash". SSD's can be completely powered down when not in use, are a COTS solution, would require little if any software development to use, and, in the case that someone else presented of a user deleting their account, the space used could be easily recovered to use for other data. I'd have to say that the Blu-ray idea is just a proof-of-concept more than a production-level solution.

Re:Write once?

[Anonymous Coward]

Because who doesn't want to smash some Blu-rays for fun?

Re:Write once?

[wiredlogic]

When I started at Kodak in 2000 they were ramping down development on their DOTS product for Digital Optical Tape Storage. It used optical film spooled in a cartridge in an automated mini-lab type machine to process data stored on the film with exposure from LEDs. Each cartridge was 1.2TB and has a 100 year shelf life. For long term, write-once archival storage it is very cost competitive against magnetic tape but they just gave up on it. The cool thing is that it looks like a company acquired the patents and is going to bring it to market after all these years.

Re:Write once?

[chmod a x mojo]

It's pretty permanent if the disc is made of durable materials.

You do realize you are talking about a company here in America right? If they finally decide to do anything, they will decide to buy the cheapest shit they can shovel in... all in the name of the almighty bottom line.

Re:Write once?

[Anonymous Coward]

Perhaps their cold storage isn't as cold as you imagine. The optical disks are much better at random seeking to pull out the occasional data asset from the jukebox. You don't have to rewind and seek the 2TB tape to fetch a record in the middle...

Re:Write once?

[Anonymous Coward]

in addition, blu-ray discs apparently have a lifespan of 100 to 150 years
the auto-changer / drive will fail long before that
the data is still on the disc so, at worst, build a new auto-changer / drive

current hard-drives are much trickier to get lasting 100 years at a decent price
i suspect drives coming out today won't last past 10 years. 20 years if you're very lucky

Re:Write once?

[CTachyon]

Anyone know if these burners are write-once drives?

If so, it pretty much guarantees that Facebook keeps a copy of your stuff forever, even if you "delete" it.

Where I work, we use large-scale tape backup (complete with robots), but tapes are so crappy that you basically have to treat them as write-once media anyway, so you have the same problems. (And tape drives are a consumable, but that's another story.) We solved this by encrypting each backup batch with a unique symmetric crypto key, and when a backup expires a cron job throws away the crypto key and marks the batch as "deleted" in our tape index. If all the batches present on a given tape end up deleted, only then do we bother to recall the physical tape from off-site storage and throw it out.

Has the bonus that we don't have to trust the security of our off-site storage provider.

Re:Write once?

[LordLimecat]

Except that a single $25 LTO4 tape will beat the crap out of those 10 BDs in terms of price, performance, convenience, and longevity.

Re:Write once?

[LordLimecat]

LTO5 tapes hold 1.5TB native and costs $25. Im really not clear why you'd use something ~4 times as expensive.

Re:Write once?

[LordLimecat]

I wasnt able to find any information on this; my understanding was that optical media such as DVD, CD, BD used inks that over time could deteriorate.

Re:Write once?

[LordLimecat]

Why has the whole world gone crazy, with people suggesting SSDs and optical to do what tape was designed to?

I mean if you LIKE paying $0.75/GB instead of $0.01/GB, sure use SSDs.

Re:Write once?

[sootman]

Maybe because WHOOSH?

Re:Write once?

[Grishnakh]

I don't follow. How is this any better than using LTO6 tapes? If the government needs to access the data, you can send them the tape too.

Re:Write once?

[bored]

T10000 tapes are the same size and store 5TB uncompressed

Now see you missed one of the cool things about the "enterprise" drives (vs LTO). Those T1k carts you purchased last year that stored 5TB, can now magically store 8.5 TB uncompressed with the latest T1kD tape drive.

Spend a few tens of thousands on new drives increases the capacity of your existing media investment by 60%, not bad if you happen to have a exabyte of capacity.

Re:Write once?

[Guspaz]

I never said that the bluray discs were more economical, only that the GP's 36GB figure didn't make sense. I'd also argue that while tape would beat optical in terms of price, it's questionable if it would win in terms of performance, convenience, or longevity. Let me discuss each:

Performance
Tape wins on pure transfer rates for a single reader. LTO-4 does 120MB/s reads, while a 16x BluRay drive will top out at 72MB/s. The performance gap is small at that level, although bigger tapes would widen it (LTO-6 does 160MB/s). What's not clear is what the actual transfer rates would be in practice; Facebook's solution uses six drives in a single rack, while tape libraries also use multiple drives. On the other hand, the gap in random access times is huge. LTO-4 has an average random access time of about a minute, while a bluray disc is about one eighth of a second. Which performance metric (access times versus throughput) is more important depends entirely on the use case.

Convenience
If you're in an automated tape or bluray library, convenience is irrelevant. If you're talking about using the media outside of that system, you'll find that a bluray disc can be read by a huge variety of computers equipped with bluray drives, while tape drives are virtually unseen outside of enterprise or the data center. If you're talking about pure density, Facebook's 1PB system claims to fit in a single rack, and looking at some commercial tape libraries it seems to me like they also need around a rack to do 1PB. Of course, if storage density is your goal regardless of cost or power usage, SuperMicro's more insane servers will do up to 4.3 PB in a single 40U rack.

Longevity
Manufacturers of LTO tape advertise longevity of up to 30 years. Estimates of the longevity of bluray media is 50-100 years, with special archival discs (like those from Milleniata) advertising well over 1000.

So, while I'm not actually convinced that Facebooks bluray plans make sense, tape isn't as superior as everybody automatically assumes it is. Personally, I'd go for an HDD-based storage cloud. Longevity if properly designed and maintained is indefinite, cost is similar to tape, performance is vastly superior to either. And if you're designing the system with cold storage in mind, power usage could be quite low (shut hardware off, turning on to check health periodically). In other words, build a reliable system out of unreliable hardware by assuming failures will happen.

Re:Write once?

[LordLimecat]

RE Performance:
I should have quoted LTO5, since they are also $25; their "native" speed is 140MB/s, and they boast ~2:1 compression which makes that 280MB/s on average. Thats roughly 4 times the speed of BD. You're right about random access, but luckily random access is about the least important factor when it comes to archival storage; sequential is far more relevant.

If you're in an automated tape or bluray library

I wasnt aware that those existed, or that there were mature backup suites for managing them.

Estimates of the longevity of bluray media is 50-100 years,

Advertising is great. Tape has actually been around for 50 years; Bluray has barely been around for 10. We know that magnetic media tends to be pretty darn stable, and we know that dye-based media (CD, DVD, BD) tend to deteriorate pretty quickly. I would be hesitant to trust anything optical for any sort of vital backups. Theyre also a pain to verify given the terrible performance compared to tape.

Youre right that tape isnt automatically superior, but there are a ton of reasons that its the de facto standard for enterprise backup.

Re:Write once?

[kheldan]

You a very funny guy, I kill you last.

Re:Write once?

[red crab]

Just think how much jobs this exercise could create - the Blu-Ray Disc Management Department.

Re:Write once?

[nabsltd]

Now see you missed one of the cool things about the "enterprise" drives (vs LTO). Those T1k carts you purchased last year that stored 5TB, can now magically store 8.5 TB uncompressed with the latest T1kD tape drive.

I forgot about that, despite the fact that we have 8 of the -D drives on order right now.

Re:Write once?

[nabsltd]

LTO5 tapes hold 1.5TB native and costs $25. Im really not clear why you'd use something ~4 times as expensive.

First, the price/TB is only about three times as much right now.

Second, a new tape drive will drop that to about twice as much, although the restore/re-write will take a while to gain that advantage.

Third, we have a fixed sized tape silo, and the investment to upgrade and increase slot capacity would be far more than spending more on the higher density tapes. Since much of what we store doesn't compress well (the data formats are already compressed), we really need as much storage per slot as possible. We're at about 4PB (native capacity) on the "used" tapes, and climbing at 25TB/day.

Re:Write once?

[mlts]

If HVD actually made it into mass production, it would be a game-changer. However, it would have to be a different diameter than CDs/DVDs/BD media... that or else be able to read the previous media types.

I remember when Tamarak out of Austin was promising a similar product, and they died with not even a whimper in the early 1990s.

Wonder what keeps holographic tech from hitting the market.

Re:Write once?

[CTachyon]

Hmm, how do you backup the crypto keys?

In short: turtles all the way down.

Not 100% sure, I don't work on that system, but I strongly suspect it's by encrypting the crypto keys with a master symmetric key and replicating/backing up the encrypted ball-o-keys as needed. The master key itself lives in an HSM; backups of the HSM are handled with the usual HSM approach of "M of N physical smartcards".

Re:Write once?

[LordLimecat]

100gb BDs are ~$50 each on amazon, while LTO5s are $25 each. Thats 30x the cost, 1/4 the speed, and no real archival track record to speak of. I just dont see the rationale.

Youre also missing that LTO5 is 1.5TB native, and in reality will hit 2-3TB.

Finally a demonstration

[cold fjord]

After reading about it for so many years now we finally get to see a demonstration of the 1,000 squirrels versus 1 ox in a demonstration. Pull!

Re:Finally a demonstration

[NotQuiteReal]

I predict the squirrels win. The one ox will eventually die, but a thousand squirrels is a viable population.

Tape?

[jythie]

I guess tape just isn't sexy anymore.

For cold storage it is still pretty hard to beat, but I have noticed a lot of tech companies have blinders regarding 'stodgy' technology.

Re:Tape?

[Big Hairy Ian]

Depends what you do with it :)

Re:Tape?

[jedidiah]

> Besides the difficulty of dealing with 174 bluray disks instead of 1 tape... You have to wonder about the reliability of those disks sitting around on a shelf for ten years..

Some of us don't have to wonder... at least not for CD or DVD.

As far as BDs go, I will get back to you in a couple years.

Re:Tape?

[cdrudge]

How soon do you need the "cold stored" data when you do need it? Random accessing one bit of information on a 1-1/2TB tape sucks. A stack of 1 BDXL discs holds the same amount of uncompressed data in less space, has almost instant access (relative to a tape drive) once the disc is loaded in a drive, and multiple discs could be loaded in multiple drives to increase simultaneous accessibility.

Re:Tape?

[swb]

How is the data stored on the media is part of the question. Is it random files dumped to the media or is it a larger database or disk image file which contains the data you want? Do I need to pull a 512 GB image file to get a 10k JPG, or can I pull the jpg directly?

Either way you will need a catalog to identify what media to use and where on that media to look. If you need to pull large blocks of data to a hot filesystem to get at a database record or to extract a smaller file, tape seems like it has a slight advantage. If its just a raw filesystem dump of files, discs sound faster, but not much.

Re:Tape?

[AmiMoJo]

Since we can assume they have knowledgeable storage specialists working for them there must be some interesting and novel reason why they are using BluRay. TFA doesn't really say, but it would be fascinating to know.

Re:Tape?

[thetoadwarrior]

What if people want something at the end of the tape? Tape is not only slow but jumping around sections isn't really possible.

Re:Tape?

[avandesande]

Tell me more of this time when tape was sexy ;-)

Re:Tape?

[LordLimecat]

Problems with BD:
  * Price
  * throughput speed (like if you wanted to make a copy)
  * Reliability-- we know how long tape lasts because its been around forever. We also know that optical tends to be piss-poor for archival.
  * its write speed sucks

Re:Tape?

[Anonymous Coward]

What if people want something at the end of the tape? Tape is not only slow but jumping around sections isn't really possible.

It takes maybe a minute or two to seek to the end of an LTO tape. The tapes are written in serpentine fashion. For instance, LTO-6 tapes have 136 wraps. The beginning of even wraps are all at beginning of tape. The beginning of odd wraps are all at end of tape.

Re:Tape?

[frank_adrian314159]

I checked with my old tape drive. Here's what she said:

It was years ago... decades, really. Back in a time when computers were big. I mean really, really big. Back when your computer didn't sit around the house! No! You designed the house around the computer!

And big! I can tell you about big! They were making movies that had computers in them! And how did you know that the large hulking box in the corner was a computer? Because it had a tape drive like spinning back and forth. Every movie that had a computer in it had one or more of us tape drives included...

And then... Well, I guess times changed. Magnetic tapes became passe. We weren't seen as a young up-and-comers anymore. We just didn't get the parts in the pictures anymore, those spots going to networking gear or some other assortment of blinking lights. And those young stars were getting so tiny, so slender... so new. And what use did the industry have for some old tape drive? I ask you, what use?

But I'll tell you one thing: I was big. I am still big! It's the computers that are small...

Re:Tape?

[Anonymous Coward]

It's cold storage - no one cares how long it takes (cf Amazon Glacier).

Reliability should be solved with something like M-Disc.

Your other points are spot on.

Reduce remaining 50% of the cost and 20% of energy

[Trachman]

Make it official by signing formal agreement with NSA, which also keeps backups of certain information. Why duplicate and waste energy?

Re:Out of touch

[dugancent]

Here they are at 50 cents/disc.

http://www.amazon.com/Optical-...

Lots of redundant data

[Rik Sweeney]

It designed the system to store data that hardly ever needs to be accessed

So that will be several million inactive profiles. I hope they've made their solution scalable, pretty soon they'll be storing 75% of their current profiles on those discs.

Re:Lots of redundant data

[perryizgr8]

they have a billion+ active profiles. i'm sure they dont have 4 billion total profiles, as ur post implies.

Re:Out of touch

[MightyYar]

Ah, nevermind. I saw the $20 per 100 price on Google shopping. But when I cilck on it, they are just DVD-Rs. Blu Ray seems to be $1, which puts them in a similar price/gig with hard drives. Mods, kill my original post please! :)

Longevity will be an issue

[Marxist Hacker 42]

NONE of those solutions, including the current one, have been tested for longevity.

I went a year between my honeymoon and getting pictures off of my 1st gen digital camera, stored in flash memory. About half were corrupt.

Re:Longevity will be an issue

[mlts]

I've yet to find a single media solution that has stood the test of time. Yes, I might be able to pull data from a tape from the 1990s, or a burned CD from 1998... but I wouldn't want to bet my data on it. Long term, the only way to do things is archive data in a format that detects (and corrects) errors (I use WinRAR, but .PAR archives work as well) and keep moving them forward in media.

Even cloud storage is unreliable. I have had sync errors completely flatten my TC volumes stored on DropBox, and restoring from Amazon Glacier is doable... but is something I have as an absolute last resort.

Re:Longevity will be an issue

[Anonymous Coward]

For that first year, he was probably trying fucking a bit harder rather than worrying about data recovery. Priorities, lonely internet dude, priorities.

Re:Longevity will be an issue

[jedidiah]

> I went a year between my honeymoon and getting pictures off of my 1st gen digital camera, stored in flash memory. About half were corrupt.

I don't even think the cheap floppies from Microcenter back in the day were that bad.

OTOH, I have plenty of optical disks sitting around in various states of neglect. I even use some of them on ocassion. I am sure I am not the only one.

I probably have a DVD burned from stuff taken off of my first digital camera that I could generate my own anecdote from. Although I prefer spinning rust for my "cold storage".

Re:Longevity will be an issue

[GrumpySteen]

I've yet to find a single media solution that has stood the test of time.

Clay tablets. Tested and proven for 5,000+ years and counting.

Space required for storage may be an issue, though.

Re:Longevity will be an issue

[hawguy]

NONE of those solutions, including the current one, have been tested for longevity.

I went a year between my honeymoon and getting pictures off of my 1st gen digital camera, stored in flash memory. About half were corrupt.

Their current solution (spinning disks) has been tested for longevity -- as long as they keep replacing failed disks (and migrating data to new storage arrays as technology obsoletes the old ones), data on spinning disks can be kept alive for as long as they can pay for it.

Re:Longevity will be an issue

[Immerman]

HD-Rosetta (http://en.wikipedia.org/wiki/HD-Rosetta) should be at least as stable (1000-10,000 years), has a much better data density, and retrieving data is almost as technology independent (you'll need only a powerful magnifying glass or electron microscope, depending on desired resolution). The question is, are you willing to pay top dollar for modern stone-age technology?

Re:Longevity will be an issue

[Immerman]

>I don't even think the cheap floppies from Microcenter back in the day were that bad.

I've got a few stories...

More to the point though magnetic tape is a relatively stable medium - keep it away from magnets and excessive moisture and it'll last nigh on indefinitely. And floppy discs are just a topological variation on the technology. Even hard discs aren't really that different, just a little more structural integrity and precision engineering to allow for finer magnetic detail.

Flash though is inherently unstable - you're storing your data as charges on microscopic capacitors, and if you fail to refresh the charges faster than leakage currents deplete them, bye-bye data. The closest other technology that I can think of is DRAM, and that requires refresh cycles several orders of magnitude more frequent.

Re:Longevity will be an issue

[Immerman]

NONE of those solutions, including the current one, have been tested for longevity.

I went a year between my honeymoon and getting pictures off of my 1st gen digital camera, stored in flash memory. About half were corrupt.

Your problem likely has little to do with longevity of Flash itself, in fact it's possible that your flash was still in near-perfect condition.

It's important to understand though that flash is a fundamentally different technology than anything else used for long-term data storage. Under ideal environmental conditions magnetic or optical media will last almost indefinitely (organic-dye CD/DVD-Rs notwithstanding). Flash won't. It's far more comparable to dynamic RAM - your data is stored as charges on microscopic capacitors with finite discharge times - if you don't "top off" the charge on a regular basis eventually all you have is a bunch of empty capacitors. The time between necessary refreshes is far longer for flash than DRAM, but the principle is the same: remove power and your data goes away. That it takes months or years instead of seconds changes the use cases, but not the fundamental instability.

Re:Longevity will be an issue

[CastrTroy]

Floppies were terrible in the sense that they could be completely ruined by putting them in a defective drive. I guess CDs and other optical medium could have this problem, but I haven't seen a on optical drive that ruined discs, since nothing is supposed to even come close to touching the disc. But I have seen plenty of floppies, zip disks, tapes, and other removable magnetic media destroyed by using a defective drive.

Re:Longevity will be an issue

[AmiMoJo]

I have archival grade CDs burnt on 1st generation drives at 2x that are readable. They were damn expensive though.

Archival grade BluRay discs are not too pricey though, because BD-R is basically magneto optical and thus all you need to do to be archival grade is use quality materials and glue. No fancy chemicals. Long term they should be fine if looked after.

Re:Longevity will be an issue

[Kjella]

I've yet to find a single media solution that has stood the test of time. Yes, I might be able to pull data from a tape from the 1990s, or a burned CD from 1998... but I wouldn't want to bet my data on it. Long term, the only way to do things is archive data in a format that detects (and corrects) errors (I use WinRAR, but .PAR archives work as well) and keep moving them forward in media.

As long as there's someone there to look after the process, but what if instead of the box of family photos in the attic you find an unreadable backup CD labeled "family photos"? The danger is that you ignore or neglect it for a little while and you can't get it back. Having something you can put in a box for 50 years and pull out at the nursing home to reminisce with would be better. And if there's ever a WWIII and people have more important things to think about for 10+ years (war and rebuilding) it would be nice if we didn't lose our entire history.

I'd still like to have something that I could be really sure that yes, this would be easily readable in consumer equipment a long time from now. I wouldn't need to have my own writer, but if you could have something etched that'd last for say 100 years and play in a BluRay player that'd be a good start.

Re:Longevity will be an issue

[Hillgiant]

http://dresdencodak.com/2009/0...

Been done.

Re:Longevity will be an issue

[TubeSteak]

I've yet to find a single media solution that has stood the test of time.

http://www.mdisc.com/what-is-mdisc/

This is a product that is shipping right now.
The discs are still expensive like BD-Rs when they first came out, but with only the capacity of a DVD-R.

Re:Longevity will be an issue

[Algae_94]

You're saying that for an entire year every spare moment of his time was spent fucking his wife? He could have at least tried to copy a couple photos of a camera sooner.

Re:Longevity will be an issue

[Anonymous Coward]

They're working on a bluray version, it's even supposed to be shipping soon. Any idea on when it finally comes out?

Tape?

[bored]

Sounds like what they really want is tape..

Besides the difficulty of dealing with 174 bluray disks instead of 1 tape... You have to wonder about the reliability of those disks sitting around on a shelf for ten years..

Oh, and you can write said tapes at 500+ MByte/sec.

Plus, tape is well understood, and there are tons of media management applications that track whats on the tape, when it expires, where its located, what encryption keys are used to decrypt it.. Basically 40 years of data management infrastructure.

News flash

[TheloniousToady]

Though I can imagine that Blue-rays may be economical for cold storage in some sort of jukebox format, it's hard to imagine how flash could be, either now or in the forseeable future. Flash storage currently is significantly more expensive than hard drive storage (ask anyone who's bought an SSD lately), and it's unlikely to get much cheaper due to fundamental limitations on the size of circuitry needed to hold enough charge to store data reliably.

Re:News flash

[Sockatume]

Think cheap CF cards, not SSDs. The read-write performance can probably afford to be terrible.

Re:News flash

[Immerman]

For cold storage SSDs are probably the wrong benchmark to compare to, being generally optimized for performance rater than price. Try something closer to SD cards. You can easily get a 64GB card for $35, so about $0.55/GB. Still more expensive than HDs for now, but a lot less than SSDs, and far more physically compact(which is another very real consideration it's you're archiving petabytes per day) Just how may SD cards do you suppose can be stored in the space of one 4TB hard drive? A heck of a lot more than the 63 required to get the same capacity I'm sure.

Of course there's still the fact that flash, like DRAM, must get regularly refreshed to maintain data integrity - leave it unpowered for a couple years and it's liable to have erased itself for your convenience. THAT in my mind would make it unsuitable for cold storage, but then again maybe Facebook's digital attic is a lot "warmer" than mine.

Re:News flash

[TheloniousToady]

FWIW, I did a little quick research on Amazon and came up with the dollars/GB of various media as shown below:

Compact Flash: $1.06 / GB
SSD: $0.68 / GB
HDD: $0.04 / GB
Blu-ray: $0.04 / GB
DVD: $0.08 / GB
Data tape: $0.02 / GB

This suggests that flash media would need to come down in cost by more than a factor of 20 to be competitive with HDDs and cheaper media. Also, Compact Flash seemed to be more expensive per GB than SSDs.

Although flash prices may drop, other media likely will also, so a relative drop by a factor of 20 seems unlikely. Factors other than cost may be a consideration, but if I were running things at Facebook, it would be pretty hard to pay 20x as much per GB just to save space.

Re:Reduce remaining 50% of the cost and 20% of ene

[Marxist Hacker 42]

Because Obama has realized the cost of storage, and the Mighty Pen of the Executive has dictated that the federal government won't spend money on that anymore, and outsource the job to the common carriers instead, who will then be required to provide exactly the same information upon request.

Re:Out of touch

[mlts]

I've been using the Optical Quantums that dugancent mentioned for a year now, and have had very good luck with them. They are about 50 cents per.

They are slow blanks, burning at 4X, but you get what you pay for, and for large backups running in the background, they are good enough. I've saved off terabytes of data using WinRAR split volumes (with recovery archives just in case) and Nero SecureDisk burning, which makes it easy to check integrity of disks before starting a restore.

I'm surprised it beats LTO-6

[hawguy]

LTO-6 holds 2.5TB/tape (native, not compressed), so it's more space dense than Blu-ray since a single tape can replace fifty 50GB bluray disks. A 1 petabyte cabinet would only need 400 tapes, and LTO tape libraries of that size are readily available off the shelf - plus the software to manage it is also off the shelf.

Cost-wise, the tapes and disks are around the same, branded dual-layer blurays seem to cost $1 - $2, and LTO-6 tapes are around $60.

The only advantage I can see for blu-rays is in random access performance, but for a rarely used cold archive system, you'd think that wouldn't matter.

Re:I'm surprised it beats LTO-6

[mlts]

I'm going into tinfoil hat territory, but I wonder if there is some advance in BD storage that FB is assuming, but the average person does not know about. If BDXL disks drop from $45.00 per disk to $1 a disk or even $2 a disk, that would change the game. Similarly, Sony/Panasonic's Blu-Ray successor that stores 300GB per disk would also be a big thing, should each disk be priced at a reasonable amount.

Re:I'm surprised it beats LTO-6

[hawguy]

I'm going into tinfoil hat territory, but I wonder if there is some advance in BD storage that FB is assuming, but the average person does not know about. If BDXL disks drop from $45.00 per disk to $1 a disk or even $2 a disk, that would change the game. Similarly, Sony/Panasonic's Blu-Ray successor that stores 300GB per disk would also be a big thing, should each disk be priced at a reasonable amount.

LTO technology keeps advancing too - LTO-7 will store 6.4TB natively on a tape.

Re:I'm surprised it beats LTO-6

[Urkki]

So what is average/worst case access time for that tape cabinet? For BD, I don't know at all, but I guess for BD it is some tens of secs including disk change and spin-up. Perhaps much less with hardware optimized for this.

If that data needs to be accessed from web, then I think some tens of seconds maximum acceptable.

Re:I'm surprised it beats LTO-6

[hawguy]

So what is average/worst case access time for that tape cabinet? For BD, I don't know at all, but I guess for BD it is some tens of secs including disk change and spin-up. Perhaps much less with hardware optimized for this.

If that data needs to be accessed from web, then I think some tens of seconds maximum acceptable.

As I said, random access might be faster with Blu-ray, but they don't seem to be interested in random access:

It designed the system to store data that hardly ever needs to be accessed, or for so-called 'cold storage' (think duplicates of users' photos and videos that it keeps for backup)

With only 16 bluray drives in the system, it wouldn't stand up to much concurrent access from the web anyway.

Re:I'm surprised it beats LTO-6

[bill_mcgonigle]

The only advantage I can see for blu-rays is in random access performance, but for a rarely used cold archive system, you'd think that wouldn't matter.

One might assume they want to re-use the same technology further up the stack as well.

Just yesterday, a friend wanted to see a picture I took of some shelves I built in my greenhouse (yes dear reader, it's time to start prepping for the food you'll eat in September). I had a picture on FB from a couple years ago. The thumbnails loaded quickly, but bringing up the full-resolution picture caused a page load time of nearly 30 seconds. Nobody had probably touched that file in 2 years, so it was being pulled off of _some_ HSM tier that's not #1 or #2. I don't know what their current implementation is, but they seem to have something in place already (which only makes sense) that's not quite as online as the article infers.

Re:I'm surprised it beats LTO-6

[WuphonsReach]

Call me when LTO-6 drives are under $300 and the tapes are under $20 for 2TB of storage.

Right now, it's $2700-$3300 for LTO-6 tape drives and $65-$75 per tape.

Doesn't add up

[michaelmalak]

TFA says 10,000 discs for 1PB, which would be 100GB/disc. But 100GB discs are $40 each (50GB discs are $2 each, and 25GB discs are $1 each). Unless they're factoring in 2x data compression the way the tape people do.

Re:Doesn't add up

[Guspaz]

100GB discs are $40 each if you're buying one of them. But what if you go to the disc manufacture and order 10,000 of them? I think you'd be able to get them a wee bit cheaper.

Re:No tape?

[mlts]

For a drive + 50 BD-R disks per TB, I'm looking at a C-note for the drive and $25 for each terabyte after that.

For a modern tape drive, I'm looking at $3500 for the drive and $65 per 2.5 TB, native capacity.

This also doesn't include hardware and software. For the LTO-6 drive, I need a dedicated server with a SAS card and a high end backup program. For Blu-Ray... it can be used, albeit slowly, with a USB 2.0 connection, but works decently with eSATA or USB 3.0.

For the big stuff, the relatively cheap price per TB of the LTO-6 drive is useful. However, not everyone can spend about $6000 for the drive, I/O card, and a decent server that can run it.

Flash as backing store for disk?!

[jeffb (2.718)]

Is flash really in a position to fill the "slower and cheaper than disk" slot, as well as the well-known "faster and more expensive than disk" slot? That's very interesting if true. Even considering power costs, I didn't think we were there yet.

Or is this some sort of near-line "flash jukebox", where most units are completely powered down most of the time?

its not the storage power thats a concern

[nimbius]

what generally kills things like SaaS providers and online content is not the cost to back up the data. tapes could be arguably just as cheap and once the data is stored, the medium is just as powerless as blu-ray. Time-to-restoration is a very big concern. things like SAS arrays of LVM striped blu-ray disks could mitigate the issue but seeing as the machinery is the same pick-and-stick model used with tape robots, throughput is going to suffer. the definition of 'cold' also comes to mind as most cold storage is one-and-done. the next interaction it sees is when its boxed and shipped to some cave in the midwest where it drives profit for some disaster recovery service like iron mountain.

what about the halflife of inks? consumer cyanic ink cd-r is supposedly OK for 10 years, although ive seen laminate degradation in less than that personally. will a disk, with arguably fewer moving parts than a tape, be able to cope with stresses to its laminates as its routinely retrieved and re-read at a data-backup frequency?

Re:its not the storage power thats a concern

[Immerman]

>what about the halflife of inks?

Indeed, I've seen data lost in under a year in impeccably cared for discs. Which is why, ironically enough, I end up using rewritable discs for archival purposes - the phase-change crystal is unlikely to change phase short of a fire that would destroy the pretty much anything anyway, so as long as the disc doesn't delaminate I'm good to go.

better solution

[Gravis Zero]

if it's so seldom accessed that it has to be moved to an optical disc, email the person and ask if they want to keep it and if they dont respond or say no, just delete it. if you have forgotten all about using your facebook account, it's unlikely that you want it to be on record for all time.

1993 Datamation called...

[3dr]

...and it wants its HSM back!

Hierarchical storage systems, and hierarchical storage managers (HSMs) have been migrating lesser-used data to cheaper (and sometimes offline) storage for decades. So what's new about it, the use of contemporary yet inefficient Blueray discs?

Re:Mainframe HSM reinvented :)

[mlts]

Even Windows had working HSM until 2008 R2 killed it. It was useful when disk was expensive -- just have a drive that you tossed your junk onto, and the OS moved it to tape.

With the next rev of USB, I wonder if it would be possible to have a LTO-6 drive usable on most machines without a Thunderbolt connection or a SAS card.

For the big boys, HSM + LTO6 + a decent tape library + some method of backing/duplicating stored tapes is the idea solution. It would be nice if this technology could wind up in the SMB world, where optical (or just external HDDs) is king now.

Re:Mainframe HSM reinvented :)

[stox]

That would be wonderful. I have yet to see a decent open source HSM implementation.

Which data

[fulldecent]

>> to store data that hardly ever needs to be accessed

In other words, data that you think you are "deleting"

We did this with financial data

[geekoid]

in 1996. Not clue ray, but write ones to dick for older account not likely to be access. So I"m not sure what's cutting edge about that.

Optical storage is dead

[funky_vibes]

When hdds are at 4T and tapes are at 6.4T it sounds ridiculous to go back to optical technologies which seem to have reached a hard limit in scalability over a decade ago.

Sounds to me like a bunch of youngsters messing around with storage robotics for fun.

This is literally true

[Burz]

Google+ recently created an account for me (without my permission) when I signed onto Youtube recently. When I accessed Google+ to get rid of it, the thing stepped me through a wizard that wanted to link me up with people I haven't conversed with for many years! I delete all mail when downloading it from gmail, and I have Google search's web history deleted and turned off.

Accessing Google generates a permanent dossier on you.

Re:This is literally true

[SirCowMan]

Facebook does the same thing, but it is not necessarily from archives of your emails etc., it may just be these suggested contacts have you on their android phone ... Or their own Gmail account. Your data may very well have been deleted, but you still exist as a sort of shadow account.

Re:No tape?

[Algae_94]

I think Facebook (The topic of this article) can afford some tape drives and servers. The scale of their data is such that drive costs are not much of a factor, media cost per GB and energy consumption would be bigger issues.

Sounds similar to Sony's offering for broadcasters

[dhaen]

http://www.sony.co.uk/pro/prod... From desk models to Petasite. It uses cartridges of blu-ray discs, UDF.

Re:No tape?

[LordLimecat]

If you want the best deal on LTO, you always use one generation back. LTO5 is far far better in price / storage-- 1.5TB native for $25, and drives for $1500 (autoloaders probably for $3000).

Re:No tape?

[bored]

However, not everyone can spend about $6000 for the drive

Vs hiring a bunch of engineers to build an optical jukebox, which is what happened in the article in question...

I don't know, i'm betting you could have bought a really big tape library with a lot of tapes/drives for the price it cost facebook to build that thing using $100k+/year engineers.

160 x BD-R 25GB discs per 4TB HDD ?

[DJRikki]

No matter what way you spin that (no pun intended) it doesnt sound all that clever. The physical space to store 160 discs would be far greater than to store a disconnected HDD.

Re:No tape?

[bored]

BTW: LTO6 half height internal tape drive (aka normal 5 1/4" drive bay) from CDW today is $2242.

http://www.cdw.com/shop/produc...

Re:No tape?

[mlts]

That is good info to know. A SAS card is a couple C-notes, so it wouldn't be too expensive to build a server around that pretty easily.

With motherboards starting to come out with Thunderbolt already present, coupled with LTO-6 drives that use that interface, even a SAS card might not be a must. I even wonder if a Mac Mini might be able to drive an external Thunderbolt tape drive.

Not to say that optical has its place -- it is cheap, but if one can pay for a LTO-6 drive, one ends up getting a lot more bang per media, more compression, and hardware level AES encryption via SPIN/SPOUT.

Economics?

[stoatwblr]

I am surprised. Even using the cheapest BRs and assuming bulk discounts, they are still way more expensive than LTO-6 or T10000D tapes (and the drives, when you're factoring in 1000 tapes per drive on average).

They're also a shedload bulkier and nowhere near as robust/long-lived.

Re:Good luck reading that data in 3 years time

[shimpei]

The failure rate on dye-based writeable optical disk based storage is horrific. There is reason to think that foil based CDs, DVDs and Bluray disks- the ones you buy with films and music pre-recorded, could last an extraordinary age if well manufactured and carefully stored, but the write-once disks are a very different technology indeed.

The organic dye used on CD-R and DVD-R has a durability problem because it is susceptible to light. BD-R uses inorganic dye, which is not susceptible to light. Completely different ballgame.

And then again, light is pretty much a non-issue in data centers because the discs will be operating inside sealed servers anyway.