Slashdot Mirror

You might find http://storagemojo.com/2007/02/26/netapp-weighs-in-on-disks/ interesting. It's not uncommon for a drive to report a fail state, then be brought back and never show issues ever again. See the section on drive resurrection. With the first fail, NetApp will treat it as a soft fail and attempt to bring the drive back online. If that also fails, it's marked as a hard fail requiring replacement. This article is from 2007 and also states "Actual MTBFs (or AFRs) of “enterprise” and “consumer” drives are pretty much the same". This isn't exactly news.

Think of the sheer amount of storage, electricity, infrastructure, personal, computing resources and so on that you would need in order to perform this feat. The numbers would be boggling and would account for a significant portion of the worldwide sales of all hard drives, tape back ups etc, etc.

Well, the internet was clocking about 21 exabytes per month in 2010. However, the overwhelming majority of that traffic is redundant; if you remove the redundancy in the data set and then compress it, you're probably looking at less than an exabyte of data over the public internet. You can reduce that further with whitelists; Traffic from Netflix, for example, is probably not going to contain super secret terrorist communications.

So let's say you can cut that down to only record the most relevant 5%. That's about 1 exabyte. How much would that cost? Well, in 2008, they guesstimated this to be about $400 million. A single stealth bomber costs about $2.1 billion; So the yearly storage costs of "the internet" is about 2 stealth bombers. -_-

So at least as far as the data storage is concerned, I think it's well within the government's budget. Now, making that data usable and analysis of it... hooo boy... that's gonna be the bitch of it. But storage? Solved.

Google did a study on consumer grade drives a while ago...

http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/archive/disk_failures.pdf

And here's how NetApp (one of those "enterprise" guys responded)...

http://storagemojo.com/2007/02/26/netapp-weighs-in-on-disks/

This tidbit known mostly to industry insiders is largely true, especially when comparing comparable drive sizes. But how storage arrays handle the respective drive type failures is what continues to perpetuate the customer perception that more expensive drives should be more reliable. One of the storage industry’s dirty secrets is that most enterprise and consumer drives are made up of largely the same components. However, their external interfaces (FC, SCSI, SAS or SATA) and most importantly their respective firmware design priorities / resulting goals play a huge role in determining enterprise vs. consumer drive behavior in the real world.

Google released a study that showed SMART to be almost useless.

It wasn't a big surprise to me, as among the hundreds of drive failures I've dealt with in my 13 years experience supporting desktops and laptops, a SMART alert was involved only once or twice.

I think it is in place to post the following information about files systems and the risk of data corruption:

(the information within this post is derived from a forum discussion with a user named "Kebabbert" so credits should go to him(/her never met him irl) for the excellent information on this post)

Regarding shortcomings in hardware RAID, here is a whole PhD dissertation showing that normal file systems are unreliable:

http://www.zdnet.com/blog/storage/ [...] t-risk/169

Dr. Prabhakaran found that ALL the file systems shared

...ad hoc failure handling and a great deal of illogical inconsistency in failure policy...such inconsistency leads to substantially different detection and recovery strategies under similar fault scenarios, resulting in unpredictable and often undesirable fault-handling strategies.

We observe little tolerance to transient failures;...none of the file systems can recover from partial disk failures, due to a lack of in-disk redundancy.

Regarding shortcomings in hardware RAID:

http://www.cs.wisc.edu/adsl/Public [...] fast08.pdf

"Detecting and recovering from data corruption requires protection techniques beyond those provided by the disk drive. In fact, basic protection schemes such as RAID [13] may also be unable to detect these problems.
..
As we discuss later, checksums do not protect against all forms of corruption"

http://www.cs.wisc.edu/adsl/Public [...] icde10.pdf

"Recent work has shown that even with sophisticated RAID protection strategies, the "right" combination of a single fault and certain repair activities (e.g., a parity scrub) can still lead to data loss [19]."

CERN discusses how their data was corrupted in spite of hardware RAID:

http://storagemojo.com/2007/09/19/ [...] -research/

Here is a whole site that only talks about the lacks and shortcomings in RAID-5:

http://www.baarf.com

Lacks and shortcomings in RAID-6:

http://kernel.org/pub/linux/kernel [...] /raid6.pdf

"The paper explains that the best RAID-6 can do is use probabilistic methods to distinguish between single and dual-disk corruption, eg. "there are 95% chances it is single-disk corruption so I am going to fix it assuming that, but there are 5% chances I am going to actually corrupt more data, I just can't tell". I wouldn't want to rely on a RAID controller that takes gambles :-)"

In other words, RAID-5 and RAID-6 are not safe at all and if you care about your data you should migrate to other solutions. In the past the disks were small and you were much less likely to run into problems. Today when the hard drives are big and RAID clusters are even bigger you are much more likely to run inte problems. Assume that there is a 0.00001% chance that you run into problems, if the hard drives are large and fast enough you will run into problems quite frequently.

What's the HDD failure rate? It's something like 2.88% (as google found, for their applications). This was back in 2007. Not sure if they've improved much over the years. In other words, SDD's are competitive and in some case (Intel's) much better than HDD's.

in that the price per gig needs to hit $1 before you start seeing acceptance for mass storage solutions from consumers.

Hmm... Hard to say, hard to say. Personally, I'm thinking more like $.10 per gig. As you mention, HDs are currently around $.05 per gig. I bought a 60gig SSD a while back, it's just not big enough - it constantly forces me to shift stuff to the HD(I LOVE symbolic links!). I can keep the OS, a few applications, and maybe a couple games on it. Performance improvements, at this point, are almost unnoticable. Personally, I think that a hybrid SSD/HD solution is currently the best idea, at least for the common user. Though I think I'd prefer 8-20 Gigs of flash cache, not 4.

Ever hear of TimeMachine on MacOS? Well, ZFS provides a similar feature to roll back changes, without the hassle of actually backing up your system. There is even a pretty Gnome UI. Of course, ZFS simplifies performing the backups, too.

Oh, did I forget to mention that despite the complexity of ZFS, the performance is stellar? Yes, it vastly out-performs traditional filesystems. Of course, that is compared to UFS on Solaris, but then again, that filesystem is on par with or faster than EXT3 (faster with logging, on par otherwise)

There is an excellent thread talking about how recent (2.6.31+) linux kernels try to report the underlying hard drive architecture (found via the OSNews comments). Alas, it looks like some of these drives are not reporting this data correctly and thus automatic adjustment (at partitioning time) is not taking place. It looks like in the future rather than trying to do detection by reported capability fdisk (and hopefully gparted) will default to sectors of 1MiB if the topology can't be found by default (unless your media is small).

Additionally, I gather that recent Fedoras will try to adjust things like LVM to match larger sectors too. Hopefully whatever is laying out LVM will also be fixed too.

Coincidentally, it looks like Oracle have a very committed dev trying to make this stuff work by default...

As far as I can see from the comparison of these FSes, BTRFS is a promising file system for Linux and is under development. Some say that it will be the ZFS of Linux or even better. I think time will say.
Some other say, now that Oracle owns Sun, Oracle can change the license of ZFS from CDDL to GPL2 and port to Linux. But porting ZFS to Linux it's another story...

Actually, high drive temperature doesn't really hurt that much. Being too cold is much worse than being too hot. Google has crunched the numbers on tens of thousands of HDDs running at different temperatures, and they found that drives that were kept coolest actually had higher failure rates. The 'sweet spot' of most stable temperatures is actually 35-45 C (95-113 F). Drives running in the 15-25 C range experienced massively higher failures than even the drives running at 50C.

Holographic Memories, Scientific American, November 1995, by Psaltis & Mok

A very good article on the principles involved in holographic storage. It focuses on solid state storage systems -- relatively large 3D lattices storage in a crystal or block of doped material. The robot that navigated its way around the lab using holographically stored images is very cool. The technology involved made use of a technique that the disks can't -- the playback beam is an image of a target, and produces an output beam from every storage cell being illuminated (up to all of them), the strength being proportional to the similarity to the target. Finding a match or closest-to is a simple matter of finding the strongest output and looking at it.

Much of the work examined was done at Stanford. They've continued developing the research, and recently announced that they holographically stored a 35 bit images "in the quantum space of a single electron" http://storagemojo.com/2009/02/03/quantum-holographic-storage/ (I assume they mean an electron's orbit). Like the 3D solid state devices before, it can stack holograms, storing two images in the same space.

In keeping with the device described in TFA, let's keep with as much current technology as possible. At 70 bit per electron, an iron atom could carry 227 bytes plus a 4 bit checksum. As iron oxide, each molecule would have 34 electrons, giving 297 bytes plus checksum. It would take 3367003367 molecules of iron oxide to carry a terabyte. What's the density of iron oxide on a disk in terms of molecules per given area? I can't find a reference, but I'm sure 3.4 billion molecules would be a tiny portion of the platter. A standard hard drive could carry enormous amounts, or disks could be made much smaller, such as the sub-inch drives (Ob/.SciFiRef) shown in Johnny Mnemonic.

As for the annoucement in TFA, there have been many such announcements from different companies for the last decade, prompting one respondent to one of the many articles to call holographic storage systems the Duke Nukem Forever of data storage. Doing a web search on various permutations of "holography" "holographic" "data" "storage" etc. produces a multitude of announcements, articles, mention of articles, second hand accountings of same, and so forth.

I've no doubt that sooner or later there will be some very expensive storage devices that a very few will be able to afford and even less make full use of. Government can afford such things and the health care industry has the cash flow plus tax write offs for business equipment. I've also no doubt the first adopters will end up stranded with expensive door stops. But these are the usual and apparently necessary steps before we can obtain the technology at more reasonable cost.

SSD Price War Begins
SSDs To Match High-end Hard Disk Prices By 2010?

Looks like some others are predicting 2 years. It's interesting to note that in January of 2007 the price per GB for a SSD was $10. It was 32GB. Today, it's $2 and change for 128GB. That's two doublings of capacity, and two halvings of price, in under two years.

Just to jump in here...you can't read parity data and invent the missing bits. You need to read the n-1 data bits + parity to work out your missing bit.

Well said. I was pretty low on the sleep when I wrote this. I should have remembered this :-)

If you really did get 1 unrecoverable error out of every 12TB read we'd have an awful lot more data loss on personal computers.

I admit I find the value counter-intuitive too but at the same time I acknowledge that given the size of data that I generally move about on my hard drive it seems plausible that these errors are beneath our ability to detect.

So if you're with me so far try bounding things on the basis of something that you would have experienced vs something you would have heard of.

For example I've never opened a word file to find a sector-sized error but since word files are generally less than 20 MB.

Even given that I shuffle hundreds of word files around on my drive. I'd still have to do it over 600,000 times to be guaranteed an URE. On the other hand if most people shuffle hundreds of small documents across their hard drive. Then it only makes the odds of
knowing someone who has had this problem around the one in one or two hundred mark. I've certainly have met people who have had corrupted files and It's difficult to discount this as a possible cause.

Now clearly bigger files represent a higher likelihood so moving up the chain lets look at the 1GB videos I routinely have on my hard drive. Currently I can't have more than about 100 or so of these on my drive but even so I'd still need to move about 12,000 of these to guarantee an error. Most of these use some form of lossy compression. So even though I am more likely to encounter a read error it also seems much less likely that I would notice. Not to mention that there are some upper layer facilities to fix problems with these files.

Let's take the biggest amount of personal data I tend to move: My RAID. I had about 1TB of data on a RAID 5 across eight 200GB discs. I moved this to a 1.5 TB RAID 5 on 4 500GB discs and then from there to a 3TB RAID 50 on 8 500GB discs.

There was only a 1 in 12 chance of any data corruption in the first move, the same in the second move. Still an 84% chance of never having a single sector hurt.

That combined with info from places like CERN

It seems a reasonable conclusion to me.

"I wish people would stop spreading this meme."

Blame google and CMU, they both found correlations and other drive vendor non-approved failure patterns - like linear increase in failures as drives age.

And right back at ya - stop spreading drive manufacturer kool-aid.

disk failures

FAIL

Similar story over at StorageMojo and Robin draws a similar conclusion.

MTBF - Infant failures about the same as discs, return rates higher
Power - Flash already near the bottom of the power curve, drives appear to have room to drop
Ruggedness - No moving parts a plus, perhaps countered by whole-block rewrites on write. Not enough data here
Noise - Flash wins, no contest

Bottom line? Not enough improvement to justify the cost, except in certain edge-conditions (like the eee PC).

The storage industry is notorious for trying to hide their price lists. Check out http://storagemojo.com/storagemojos-pricing-guide/ for street prices on storage gear. It's not all up to date, but you can get a ballpark without requesting all sorts of quotes from a reseller.

The inevitable result is a race to the bottom. Buyers will reason they might was well buy cheap, because they at least know they're saving money, rather then paying for quality and likely not getting it. I cheat. I ask the storage administrators I know. One place I know is really good, they see drives from everywhere. You get a feeling on which drives are better than others. About 5-7 years ago one manufacturer really messed up, gaining a nickname "death....". I still don't buy them today. Another I stopped buying went out of business years ago.

I know google collects these stats but getting the inside on it is tough. But here is a glimpse of what they have.

RAID-5 is EVIL! Never Ever, EVER use RAID-5, You will LOSE DATA! RAID-1 or RAID-10 only for production use.

If you are thinking about Raid-5, forget it, just stripe your drives in a RAID-0 and enjoy the performance benefits and keep frequent good backups and test your restores.

Why RAID 5 stops working in 2009
Why aren't disk reads more reliable?
End of Raid 5

finally, BAARF - Battle Against ANY Raid Five BAARF

HTH, HAND, don't cry.

Read the article and watch the video. They talk about that.

You don't actually believe that number anyway do you? MTBF doesn't mean anything. If you read the google papers on hard drives you would discover that the numbers posted don't mean anything.

Don't forget to read the information about CERN and it's data errors.

Read the article and watch the video. They talk about that.

You don't actually believe that number anyway do you? MTBF doesn't mean anything. If you read the google papers on hard drives you would discover that the numbers posted don't mean anything.

Don't forget to read the information about CERN and it's data errors.

Two sentences I would highlight from the StorageMojo article:

1) Developers, the time may not too far away when your code is measured on power efficiency.
2) Software effects will be found significant as well because widely used software affects so many systems.

This reminds me of an article here on /., about how Microsoft could become the world's greenest company with a few small changes in code to be more aggressive about using power saving modes by default. Hardware makers have been harping on about power efficiency for ages, driven largely by the mobile computing and gadgetry market. As a result, there are a lot of things that the hardware can do to save power when under-utilized or idle, even in the server market. However, those features are only useful if the software (particularly OS and firmware) takes advantage of them. The more widespread the software, the greater the impact of incremental improvements in power-efficient code. A tiny fraction times GWh of energy is still a large amount. There are whole realms of power efficiency that have yet to be tapped.

Agreed. I'd bet that the mechanical components, specifically the ball-bearings in the drive motor, are more likely to overheat and fail. In addition power-regulation/power-supply components such as large power transistors and resistors on the logic board are likely to fail.

After 5 years of solid running, a lot of hard drives begin to sound different. Guess what, thats the bearings wearing out... More intersting stuff http://storagemojo.com/?p=378

StorageMojo points out that last generation 15k drives already write over twice as fast as the article claims they do.

Add to that the points that in 10 years areal density will be up at least 100x, that write performance is only a part of total disk performance and that you'd need 75k drives to take advantage of the increase.

One can only conclude that the slow summer silly season is upon us.

1. Feature Set:
   First off, every serious NAS/SAN vendor is going to have a snapshot solution. Here are a couple other features you might need: Automatic Replication, High-Availablity / Failover, Integrated Virus Scanning, Clustering. Many of the "exciting" features listed in Sun's press releases are not even vaguely revolutionary.
2. Manageability:
   As the name "NetApp" may imply, NAS/SAN vendors often sell "appliances" that attempt to simplify many of the management concerns (e.g. monitoring, automation of backups, etc). How well they do this varies from vendor to vendor and based on what features you need.
3. Compatibility:
   Large NAS/SAN vendors have already verified that their product works with a number of 3rd-party apps and hardware. Will your old tape hardware work well with ZFS? Is SQL certified to run on ZFS? Will Sun's customer support help you if you do get things working? Likely not.
4. Performance:
   For all the business I've heard about ZFS being the "last word in file systems", the amount of actual performance data has been incredibly lacking. For example, most NFS products have published their SPEC numbers. Although these performance results are often gamed a bit, they're the current standard for NFS performance.
   The performance numbers I have seen with ZFS so far are useless (e.g. see here for someone measuring how fast ZFS can write to RAM or here for someone getting the blazing throuput of 45 mb/s). Filesystem-based iSCSI solutions (as opposed to SANs) tend to have terrible performance (this includes some of NetApp's products), so I'm a bit dubious about claims that ZFS does iSCSI faster than it does NFS.
5. Reliability:
   In addition to reliability features such as High-Availability and Disaster Recovery, how many enterprise production environments is ZFS actually running in? How many data corruption bugs are waiting to be ironed out? How mature are the repair tools (e.g. fsck)?
   Also, enterprise NAS/SANs (e.g. those of NetApp) often have a nice feature where an operation is stable once the client receives an ACK. They get this by logging pending operations to non-volatile RAM. As far as I can tell this is not possible with ZFS, which means that your applications need to be aware that operations may need to be resent to the server after crash.

Google have a great solution that focuses on the “cheap” part without compromising much the latter two. If you have not read up on the Google Filesystem, definitely take the time to. At the very least, it seems to call into question the need to shell out tens of thousands for high-end storage solutions that promise reliability in proportion to the dollar.

Robin Harris over at storagemojo.com has been blogging about flash and flash life expectancy for some time. Check it out. http://storagemojo.com/?p=281 Oh, and I'm not Robin.

After their first reallocation, drives are over 14 times more likely to fail within 60 days than drives without reallocation counts, making the critical threshold for this parameter also one.

According to this NetApp reply to an open letter on storagemojo while the electronics of the drive beyond the interface may be the same on consumer and enterprise drives, the way the firmware behaves is not. The consumer drive firmware apparently do all it can to try and read data back even if it makes the drive temporarily unavailable and trusts additional information less that enterprise firmware.

This is good reading:
http://storagemojo.com/?p=383

Short synopsis for those who don't want to read it: The rebuild process is intense enough to cause secondary failures in many more cases than you'd think. Because you haven't seen it yet is not indicative of the overall population, and sysadmins are payed to be prepared.

The rest of your post is arguable, but it's more a matter of opinion and practice than anything else.

for people who want the bottom line and not a 13 page paper. Check out Google's Disk Failure Experience.

Not sure where Apple starts supporting it: in the server or the desktop version of Mac OS X. But the signs for ZFS support are clear.

ZFS is vastly superior to ReiserFS 3/4, to ext2/3/4, to xfs, etc. It's fast, journalled, expandable on the fly, supports snapshots, checksums, redundancy, ... Among the few things that miss is builtin encryption, quotas (can be solved by "personal" filesystems) and online raid expansion. But in essence it's very cool. Most of the things that make TimeMachine look cool are built into ZFS. With the feature set of ZFS TimeMachine is not much more than a fancy GUI.

Read more at wikipedia or here (How cool is that).

Bye egghat

Don't any of you guys use Macs?

Here are five good reasons for Apple to go to ZFS:
-No more Disk Warrior. The entire data store is self-validating. No bit rot.
-No RAID controllers needed: ZFS gives you fast RAID for free. Just add drives. Why would anyone care? See #5.
-No more volumes and, therefore, no more volume management. ZFS eliminates the whole volume concept. Add a disk to your system and it joins your storage pool. More capacity. Not more management. What home user would want that?
-Continuous Data Protection out of the box. Time Machine could give you a view of your data every time you update a file.
-ITV, or whatever it is going to be called. Multi-GB files that each cost $10-20, that can't be backed up - thanks DRM! - and therefore need a cheap and highly reliable RAID. ITV, two firewire drives, ZFS and you are in business.
-Not to mention the existential pleasure of having great technology that Vista doesn't have. In fact, since consumer technology is driving the enterprise, expect ZFS on Mac to raise the bar for every OS and file system.

I suspect that Time Machine is simply the first of several beautifully designed storage utilities that we'll see on Leopard. How about automatic synchronization when you plug in an external drive? Snapshots automatically exported to .Mac? ZFS enables all kinds of coolness and I, for one, can't wait to get it on my laptop.

Read more at ZFS On Leopard: How Cool Is That? Means, Motive & Opportunity: Apple Kills the Media Center PC and the latest ZFS On Mac: Now All-But-Official.

And you heard about the native iSCSI support in Leopard, right?

Don't any of you guys use Macs?

Here are five good reasons for Apple to go to ZFS:
-No more Disk Warrior. The entire data store is self-validating. No bit rot.
-No RAID controllers needed: ZFS gives you fast RAID for free. Just add drives. Why would anyone care? See #5.
-No more volumes and, therefore, no more volume management. ZFS eliminates the whole volume concept. Add a disk to your system and it joins your storage pool. More capacity. Not more management. What home user would want that?
-Continuous Data Protection out of the box. Time Machine could give you a view of your data every time you update a file.
-ITV, or whatever it is going to be called. Multi-GB files that each cost $10-20, that can't be backed up - thanks DRM! - and therefore need a cheap and highly reliable RAID. ITV, two firewire drives, ZFS and you are in business.
-Not to mention the existential pleasure of having great technology that Vista doesn't have. In fact, since consumer technology is driving the enterprise, expect ZFS on Mac to raise the bar for every OS and file system.

I suspect that Time Machine is simply the first of several beautifully designed storage utilities that we'll see on Leopard. How about automatic synchronization when you plug in an external drive? Snapshots automatically exported to .Mac? ZFS enables all kinds of coolness and I, for one, can't wait to get it on my laptop.

Read more at ZFS On Leopard: How Cool Is That? Means, Motive & Opportunity: Apple Kills the Media Center PC and the latest ZFS On Mac: Now All-But-Official.

And you heard about the native iSCSI support in Leopard, right?

Don't any of you guys use Macs?

Here are five good reasons for Apple to go to ZFS:
-No more Disk Warrior. The entire data store is self-validating. No bit rot.
-No RAID controllers needed: ZFS gives you fast RAID for free. Just add drives. Why would anyone care? See #5.
-No more volumes and, therefore, no more volume management. ZFS eliminates the whole volume concept. Add a disk to your system and it joins your storage pool. More capacity. Not more management. What home user would want that?
-Continuous Data Protection out of the box. Time Machine could give you a view of your data every time you update a file.
-ITV, or whatever it is going to be called. Multi-GB files that each cost $10-20, that can't be backed up - thanks DRM! - and therefore need a cheap and highly reliable RAID. ITV, two firewire drives, ZFS and you are in business.
-Not to mention the existential pleasure of having great technology that Vista doesn't have. In fact, since consumer technology is driving the enterprise, expect ZFS on Mac to raise the bar for every OS and file system.

I suspect that Time Machine is simply the first of several beautifully designed storage utilities that we'll see on Leopard. How about automatic synchronization when you plug in an external drive? Snapshots automatically exported to .Mac? ZFS enables all kinds of coolness and I, for one, can't wait to get it on my laptop.

Read more at ZFS On Leopard: How Cool Is That? Means, Motive & Opportunity: Apple Kills the Media Center PC and the latest ZFS On Mac: Now All-But-Official.

And you heard about the native iSCSI support in Leopard, right?

I have to laugh about all the Technology that Sun supposedly invented. Please! Sun's original success came not from invention, but from surfing fast and hard on (Joy's) BSD and (Bechtolsheim's) commodity hardware workstation. Invent as little as possible and innovate like crazy using cheap building blocks (HW & SW) while (Scott)promo'ing and selling the hell out of the result was the original Sun recipe for success. It is a good recipe and would still be working if Scott hadn't tried to turn Sun from a technology exploiter to a technology inventor, trying to ape DEC, HP and IBM, the firms Sun vanquished in the workstation market. Look at what Sun did invent. SPARC, useful for a time but hung onto way too long. Solaris, ditto, and when finally open-sourced too little too late. Java, a great achievement, but Scott was too fearful to let it go free. And the failures in the storage business, where Scott's old bomb-throwing ways could have earned billions in high-margin revenue and turned a stodgy backwater into an industry leader. But NO -- buy a mainframe tape company for billions in cash. A brief overview of the whole sordid story here. What Scott did well was to create a culture where everyone went 100 MPH towards the current goal, and when the inevitable mid-course corrections came, turn on a dime and continue at 100 MPH. A rare and wonderful accomplishment. If only he'd left on a high note.