Slashdot Mirror

Many have pointed out the need for a better understanding of what disk failures look like in the field. Yet hardly any published work exists that provides a large-scale study of disk failures in production systems. As a first step towards closing this gap, we have analyzed disk replacement data from a number of large production systems, spanning more than 100,000 drives from at least four different vendors, including drives with SCSI, FC and SATA interfaces. Below is a summary of a few of our results.

• Large-scale installation field usage appears to differ widely from nominal datasheet MTTF conditions. The field replacement rates of systems were significantly larger than we expected based on datasheet MTTFs.

• For drives less than five years old, field replacement rates were larger than what the datasheet MTTF suggested by a factor of 2-10. For five to eight year old drives, field replacement rates were a factor of 30 higher than what the datasheet MTTF suggested.

• Changes in disk replacement rates during the first five years of the lifecycle were more dramatic than often assumed. While replacement rates are often expected to be in steady state in year 2-5 of operation (bottom of the ``bathtub curve''), we observed a continuous increase in replacement rates, starting as early as in the second year of operation.

• In our data sets, the replacement rates of SATA disks are not worse than the replacement rates of SCSI or FC disks. This may indicate that disk-independent factors, such as operating conditions, usage and environmental factors, affect replacement rates more than component specific factors. However, the only evidence we have of a bad batch of disks was found in a collection of SATA disks experiencing high media error rates. We have too little data on bad batches to estimate the relative frequency of bad batches by type of disk, although there is plenty of anecdotal evidence that bad batches are not unique to SATA disks.

• The common concern that MTTFs underrepresent infant mortality has led to the proposal of new standards that incorporate infant mortality[33]. Our findings suggest that the underrepresentation of the early onset of wear-out is a much more serious factor than underrepresentation of infant mortality and recommend to include this in new standards.

• While many have suspected that the commonly made assumption of exponentially distributed time between failures/replacements is not realistic, previous studies have not found enough evidence to prove this assumption wrong with significant statistical confidence[8]. Based on our data analysis, we are able to reject the hypothesis of exponentially distributed time between disk replacements with high confidence. We suggest that researchers and designers use field replacement data, when possible, or two parameter distributions, such as the Weibull distribution.

• We identify as the key features that distinguish the empirical distribution of time between disk replacements from the exponential distribution, higher levels of variability and decreasing hazard rates. We find that the empirical distributions are fit well by a Weibull distribution with a shape parameter between 0.7 and 0.8.

• We also present strong evidence for the existence of correlations between disk replacement interarrivals. In particular, the empirical data exhibits significant levels of autocorrelation and long-range dependence.

Obviously, then, it should have been possible to fetch that data from the DC (probably using LDAP) with AD as well. Any way I look at it, it seems they did it to prevent non-Microsoft KDCs to work with Microsoft clients. Especially seeing how the authorization data in the ticket is something as simple as the Windows SIDs for the user.

PhysRevLet is behind the times. The trend is for open access. This week, USENIX, the computing systems association and sponsor of many major conferences, is making access to all its published papers and conference proceedings free to the world. This blog has details.

You are wrong, in fact the small feature size of modern HDD's actually makes it easier in some cases as the smaller magnetic domains are harder to flip so even small changes in alignment will mean that recoverable data will be left behind.

I'm not sure that impression is accurate. Dr. McGraw gave Cornell's Association of Computer Science Undergrads a presentation about this topic. (He used the same slides as those available here: http://www.usenix.org/events/sec07/tech/#thurs . I would recommend looking at that presentation, though I don't know how different it is from the one I saw.) He was representing his company, which audits code for security problems, and one of the main points of his talk was that all this stuff being used to make (or thwart) exploits in MMORPGs are also ideas that crop up in other applications. I thought the presentation was interesting and well-balanced, and, to me, it didn't come off as an attack on WoW at all.

He did admit to us that his pro-security books weren't selling much; apparently, people much rather learn via breaking other peoples' code than fixing their own. (Figures.)

The increased sales from referencing MMORPGs or WoW might well be padding his pockets, but if it's also helping people to write more-secure code, I say go for it.

Also, the authors of Exploiting Online Games have a sample chapter available, and Usenix has a video of one of Gary McGraw's presentations on their web site.

Similar work has already been published at Usenix Security. http://www.usenix.org/events/sec07/tech/akritidis.html
Full paper is available at one of the authors' website. http://s3g.i2r.a-star.edu.sg/papers/metrowifi-usenixsec07.pdf

Not to distract from the interesting nature of the article but people really should do
some related work background research:
http://www.usenix.org/events/sec07/tech/akritidis.html

These guys showed this (and other privacy related attacks) last year at Usenix Security.

For OpenBSD specifically, well, they use a variant on blowfish for their hashing, called eksblowfish (for expensive key schedule). The key setup portion is purposefully slow; this severely limits speed of brute forcing attempts. Further, they use the salt in the key setup phase, which limits precomputation in the key schedule portion. Due the specific choices they made in designing eksblowfish, they need exactly 128 bits of salt, no more no less. Also, they are limited to 55 character passwords. Changing this limitation would require fundamental changes to the underlying blowfish algorithm, and is hence difficult to do. Essentially, they traded off the eks portion of their password hash (which makes it slow to brute force and resistant to future technological advances) for flexibility in password length and salt length. A paper detailing OpenBSDs password hashing is available from Usenix at http://www.usenix.org/events/usenix99/provos.html for free.

(As an aside, they could have done better. Of course, it is easy to say that after the fact, given people have had years to think about eksblowfish; they were then and still are now just about the best password hashing scheme one could concoct. A slight strengthening would be to use a good hash function (e.g. SHA-512) to pre-hash the password, then truncate that for use as the "password" in eksblowfish. This would allow arbitrarily long passwords, although it wouldn't allow more than 512 bits of entropy in the password, or any more entropy than eksblowfish currently takes)

For other hashing schemes, there is no point in using more salt than the size of your password hash. If your hash is only 128 bits (as is MD5), then for every salt longer than 128 bits, there is a shorter salt which collides with it. Since the real threats are dictionary and rainbow table, and attackers can take advantage of collisions, you cannot use salting to make the problem any harder than your hash length. There is a cost to longer password hashes, although it is relatively minor (and, per the argument of the OpenBSD crowd, you *ought* have an expensive hash).

On the other hand, even 64 bits of salt is quite paranoid anyway; it makes a dictionary attack a bit over 18 billion billion times harder. 128 bits is overkill already. Extreme overkill. It is far easier to just brute force the password (which salting doesn't help) than to use any precomptutation at that point. The average password has very few bits of entropy; this is a fundamental limit of passwords. It is hard to remember things with high entropy. Even long sentences don't have that much entropy, because sentences have a lot of predictability in them. Remembering enough to make Linux's 48 bit salts the weak link is pretty much impossible for your average user, and requires a special effort even for people who care.

True, but it's fair to point out that there's a bit of new evidence which counters the bathtub curve model for hard drive failure. From http://www.usenix.org/event/fast07/tech/schroeder/schroeder_html/index.html:

While replacement rates are often expected to be in steady state in year 2-5 of operation (bottom of the ``bathtub curve''), we observed a continuous increase in replacement rates, starting as early as in the second year of operation.

Wah wah wah! Grow up. You sound like a spammer.

A spammer who published a paper on automated classification of spam, and devised a neural network/information clustering technique which was shown to be even more effective than Bayesian filtering -- in fact, more effective than ANY other known content-based method at the time? Yeah, okay, chief. So tell me, what the hell have YOU been doing to combat the spam problem, aside from widesweeping, ill-advised, technically flawed, misanthropic methods?

Filter the content, not the physical source. We could beat the spammers by shutting down the whole damn Internet, but that's not a real solution. It's a solution for the simple-minded and the impatient. Who cares if a few percent of spam gets through? WE HAVE OUR FREEDOM BACK. Your attitude seems to be, "Who cares if we give up our free use of the Internet because of a few dickheads -- at least I don't have to deal with the inconvenience of spam messages."

I'm willing to put in the work to make an open, spam-free Internet a reality. How about you? Or would you prefer to just yank my network connection so you don't have to hear my "whining" any more?

We do not encrypt passwords (well, not usually). We hash them. There is nothing wrong with Blowfish crypt. It is very, very secure. Much more secure than either salted MD5 or SHA-1. However, there is really nothing wrong with using either MD5 or SHA-1 in the short term (so long as you are using a proper salt!). They will do fine. The eight character password, though, has been antiquated for about five years now. You should have switched to 10-15 characters some time ago. Really, even 15 characters is walking a thin line now.

But perhaps you would like to read more about Blowfish crypt:

http://www.usenix.org/events/usenix99/provos/provo s_html/node5.html

A better alternative is using a hash function with an adjustable cost (and good salting function with a large salt space), or you could just stop using passwords all together.

There are ways to recover Data from ram even after you remove the power. Check out this paper.

http://www.usenix.org/publications/library/proceed ings/sec96/full_papers/gutmann/

It can happen, but usually because you've tripped over a threshold. Loading up Java, for example, or taking a *little* bit more RAM and starting to swap can make a huge performance difference. And an optimized implementation of a heavily used function can make a big diffreence in specific applications: take a look at the old Squid "select" problems, as described at http://www.usenix.org/publications/login/1999-2/sq uid.html.

According to this paper http://www.usenix.org/events/fast07/tech/schroeder .html presented at FAST 2007 (a usenix conference), a disk reporting smart errors has a high probability of future failure.

The entire article is a very worthwhile read about disk failures.

The Dell key-logger hoax has probably the best decoy story to move
professional hackers/security staffers into the wrong direction, as in
May 2006, USENIX published the following research article :

"Keyboards and Covert Channels"
  by Gaurav Shah, Andres Molina and Matt Blaze , 2006-05-17
  Department of Computer and Information Science
  University of Pennsylvania
http://www.usenix.org/events/sec06/tech/shah/shah_ html/jbug-Usenix06.html

In it the authors demonstrate that todays unwarranted wire tapping NSA
activities, normally don't result in much success as serious internet
users routinely apply encryption into their communications, like IPSec
tunneling, ssh, VPN access connections, secure web-traffic https when
i.e. doing Internet banking activities.

However, secret service found a clever approach to all this, by
covertly installing a Keyboard JitterBug into your keyboard. Here's
how to secure your most trusted keyboard :

Keyboard JitterBug eavesdropping
http://crashrecovery.org/internet/#jitter

where i may add, that lock picking _ALSO_ has been the best hoax ever
on public display. Why? How many people today design their _OWN_
locksmith locks? All installed door-locks worldwide are somehow sold in
stores, hence its products and replacement keys are in the archives of
the local secret service.

Robert

I would be interested in knowing your assumptions, test streams and the hardware configuration. Several carefully executed benchmarks of real world scenarios that I have seen do not tally with what you are claiming. For instance, see Performance Evaluation of Virtualization Technologies for Server Consolidation and Diagnosing Performance Overheads in the Xen Virtual Machine Environment

I haven't read all the papers but there may be something useful from USENIX HotBots

Forget Raid , RAId Doesn't work.

Remember the Usenix paper ?
http://www.usenix.org/events/fast07/tech/schroeder .html

Just make sure your application or you know how to store data in multiple places. You'll get both a cheaper solution (less disks required, no fiber required) and a more robust solution straight no nonsense access to the data you need.

Maybe Vista now has a unified buffer cache, which makes the whole idea of "cache" rather blurry. Take this FreeBSD system for example:

Mem: 1891M Active, 5131M Inact, 240M Wired, 257M Cache, 214M Buf, 243M Free

In this case, the vast majority of "Inact" is made up of cached file data, but such cache will also be spread around "Active" (can be swapped, but would likely to be swapped back in soon after) and "Cache" (rarely used pages which can be freed quickly because they aren't "dirty"). Depending on how you define "memory use" you could say I'm using anywhere from 2.3 to 7.5G. Even these are rather blurry since the lack of memory pressure means the various lists aren't being cycled very aggressively.

They were/still are.

I think in TFA or elsewhere I've read that papers were signed about two weeks ago. Just about the time I was reading some stuff about the http://www.usenix.org/events/hotbots07/tech/full_p apers/provos/provos.pdf "The Ghost in the Browser".

Reckon the guys are using Greenbox as part of their malware tests - they run malware within a virtual machine to monitor the malwares actions.

If they continue to use the system as described in the doc to test, evaluate and thus detect malware seems to me that Greenbox is a handy tool to have.

Here's the actual paper. It's a Usenix paper.

What they're doing is straightforward, and it's much like what many virus scanners do. First, they look at web pages to see if there's anything suspicious that requires further analysis. If there is, they load the page into Internet Explorer (of course) in a virtual machine, and see if it changes its environment. The better virus scanners have been doing something like that for a few years now, running possible viruses in some kind of sandbox. Although they usually don't go all the way and run Internet Explorer in a virtual machine. (Are you allowed to do that under Microsoft's current EULA for IE 7?)

The main problem with Google's approach here is that it's after the fact. They won't notice a bad page until the next time they crawl it. Bad pages come and go so fast today that they'll always be behind. As the paper says, "Since many of the malicious URLs are too short-lived to provide statistically meaningful data, we analyzed only the URLs whose presence on the Internet lasted longer than one week."

If Google implements this, the main effect will be to push attackers into changing site names for attack sites even faster.

It's all so backward. What we need is to run most of Internet Explorer in a tightly sandboxed environment on the user's machine, so that when you close the window, any browser damage goes away. That would actually work.

Maybe security by obscurity is still considered valid by USENIX conference organizers? :)

Yes, that is a joke. It's probably due to space limitations, or they don't want it to take on a Black Hat '07 ambiance or something.

I'd be amazed if at least the best couple of papers didn't appear on the portion of usenix.org available to non-members.
http://www.usenix.org/publications/library/proceed ings/best_papers.html

BTW, the editor has made a *gasp* mistake. USENIX is a professional organization for anyone that uses a Unixy OS, not just academics. It's companion organization, SAGE, is for SysAdmins. That was almost spun off a couple of years, ago, but in the end it didn't work.

Dues are reasonable. $165/yr. for both, and there are student discounts. I know a couple of people who've been able to expense their dues.
https://db.usenix.org/cgi-bin/memb/memb.cgi?action =new

I would recommend either or both, depending upon what your doing at the moment. Look the Web site over, and form your own conclusion, of course.

Maybe security by obscurity is still considered valid by USENIX conference organizers? :)

Yes, that is a joke. It's probably due to space limitations, or they don't want it to take on a Black Hat '07 ambiance or something.

I'd be amazed if at least the best couple of papers didn't appear on the portion of usenix.org available to non-members.
http://www.usenix.org/publications/library/proceed ings/best_papers.html

BTW, the editor has made a *gasp* mistake. USENIX is a professional organization for anyone that uses a Unixy OS, not just academics. It's companion organization, SAGE, is for SysAdmins. That was almost spun off a couple of years, ago, but in the end it didn't work.

Dues are reasonable. $165/yr. for both, and there are student discounts. I know a couple of people who've been able to expense their dues.
https://db.usenix.org/cgi-bin/memb/memb.cgi?action =new

I would recommend either or both, depending upon what your doing at the moment. Look the Web site over, and form your own conclusion, of course.

The main problem is that manufacturers loudly blare the MTBF/MTTF values without telling people how long the test was done, hence they can use whatever time they like. You have to agree in the very least that even for comparison between products from different manufacturers, the MTBF is useless, for the simple fact that one manufacturer might test for 100 hours, and the other for 1000 hours...

As such an unreliable measure, the first two letter 'MTBF' stands for 'misleading'.

I didn't just say it, Carnegie Mellon University examined reliability and found 'MTBF' doesn't mean much:

http://www.usenix.org/events/fast07/tech/schroeder /schroeder_html/index.html

"mean time between failure of 5 million hours"

Didn't we just recently learn that they're pulling these numbers out of their arse, and that they're essentially useless?

Disk failures in the real world: What does an MTTF of 1,000,000 hours mean to you?

This was covered on Slashdot already.

If you're going to read Slashdot, at least fucking read it.

Aero

"Why should I replace my 5 year old drive with an identical new one? I shouldn't."

Read the FULL report ( http://www.usenix.org/events/fast07/tech/schroeder /schroeder_html/index.html )

Scroll down to "4.2 Age-dependent replacement rates"

It looks pretty clear from that graph that drive 5 years old are highly more likely fail. If you are a statistics person, thats seems pretty convincing.

Not to say you need to chuck the drives, my anecdotal evidence being I have 2 linux boxes running with 10 year old 4.3GB IBM drives, and they just work.

Except they didn't study what "people" thought, they studied "a number of large production systems, including high-performance computing sites and internet services sites" - in other words, the best case scenario in terms of user expertise.

This is handled in the paper. See this graph: http://www.usenix.org/events/fast07/tech/schroeder /schroeder_html/img14b.PNG

Unfortunately there is no big "spike"; the average replacement rate just grows and grows with time.

In other news, Carnegie Mellon researchers know more about statistics than you give them credit for; blame ComputerWorld for crappy coverage of what the paper says. If you read the paper or the abstract, the researchers actually claim the opposite of what you are suggesting, namely, that the "infant mortality effect" (bathtub curve) often claimed for hard drives isn't actually the case. See Figure 4 in the paper and Section 5 ("Statistical properties of disk failures"). The paper is online here:

http://www.usenix.org/events/fast07/tech/schroeder /schroeder_html/index.html

TFA seems surprised by SATA drives lasting as long as Fibre...why one earth would your data interface have any consequences on the drive internals?

Because drive manufacturers claim they use different hardware for the drive based on the interface. For example, a SCSI drive supposedly contains a disk designed for heavier use than an ATA drive, they aren't just the same disk with different interfaces.

• the MTTF is always much lower than the observed time to disk replacement
• SATA is not necessarily less reliable than FC and SCSI disks
• contrary to popular belief, hard drive replacement rates to not enter steady state after the first year of operation, and in fact steadily increase over time.
• early onset of wear-out has a stronger impact on replacement than infant mortality.
• they show that the common assumptions that the time between failure follows an exponential distribution, and that failures are independent, are not correct.

Plan 9 has such a central key repository. It's called Factotum, and the best description is the USENIX paper. It has been ported to other UNIX-likes by the plan9port project.

However, it's pretty much always a support service. Therefore you should expect that you'll end up on call. Personally I don't like that part, but can't deny the extra pay is nice.

It's also a field where experience is what really really matters. Which means it can be tough to break into. Certifications and degrees are nice, but it's my '5 years in the industry' which opens doors, not the other bits of paper.

However as a starting point in 'building your career', I will suggest you look at:

• ITIL - IT infrastructure library. It's something that put me off initally, as it look a bit too much like icky-yuck processes and procedures. However, I've run into a _lot_ of companies that are starting to 'buy in' to the model. That wouldn't convince me, though. What did, is it's actually a fairly good way of 'doing IT'. Not the only way by any means, but one worth looking at, if only because then you have a basis for comparison.
• SAGE Systems Administrators guild, a subdivision of Usenix.
• BCS British Computer Society
• The Practice of System and Network Administration (Paperback) - A personal favourite, this is a brilliant book, because it covers the _theory_ of systems admin.

As far as I can tell, your bits of paper serve to help you secure an interview. But the field's .... well sufficiently complicated and convoluted that your ability to learn, research and innovate are far more important. As is your ability to show you can do this.

According to:

http://www.usenix.org/events/fast07/tech/schroeder /schroeder_html/index.html

The chances of a double disk failure in a RAID 5 are significantly greater than we think. A friend of mine had a double failure just last week.

You might be interested in the other best paper award winner (in the shameless self-promotion department): TFS: A Transparent File System for Contributory Storage , by Jim Cipar, Mark Corner, and Emery Berger (Dept. of Computer Science, University of Massachusetts Amherst). Briefly, it describes how you can make all the empty space on your disk available for others to use, without affecting your own use of the disk (no performance impact, and you can still use the space if you need it).

Enjoy!

--
Emery Berger
Dept. of Computer Science
University of Massachusetts Amherst

You might be interested in the other best paper award winner (in the shameless self-promotion department): TFS: A Transparent File System for Contributory Storage , by Jim Cipar, Mark Corner, and Emery Berger (Dept. of Computer Science, University of Massachusetts Amherst). Briefly, it describes how you can make all the empty space on your disk available for others to use, without affecting your own use of the disk (no performance impact, and you can still use the space if you need it).

Enjoy!

--
Emery Berger
Dept. of Computer Science
University of Massachusetts Amherst

You might be interested in the other best paper award winner (in the shameless self-promotion department): TFS: A Transparent File System for Contributory Storage , by Jim Cipar, Mark Corner, and Emery Berger (Dept. of Computer Science, University of Massachusetts Amherst). Briefly, it describes how you can make all the empty space on your disk available for others to use, without affecting your own use of the disk (no performance impact, and you can still use the space if you need it).

Enjoy!

--
Emery Berger
Dept. of Computer Science
University of Massachusetts Amherst

In his table, under "Research Lab", Google gets a "Not really". This guy has absolutely no idea about what he's talking about. Google has some of the most cutting-edge research in the industry. They almost always have research papers published at the conferences that I attend (so does MS, but Yahoo rarely does). Here are some examples.

In his table, under "Research Lab", Google gets a "Not really". This guy has absolutely no idea about what he's talking about. Google has some of the most cutting-edge research in the industry. They almost always have research papers published at the conferences that I attend (so does MS, but Yahoo rarely does). Here are some examples.

In his table, under "Research Lab", Google gets a "Not really". This guy has absolutely no idea about what he's talking about. Google has some of the most cutting-edge research in the industry. They almost always have research papers published at the conferences that I attend (so does MS, but Yahoo rarely does). Here are some examples.

Site-wide can work. It just takes some extra tuning considerations. See e.g. http://www.usenix.org/events/lisa04/tech/blosser.h tml

The predictions were that it wasn't feasible but the evidence from multiple quarters hasn't supported that so far. I'm not sure anyone has published anything on *why* that might be yet, but it is probable that there are population size break points between which legit mail is relatively homogenous enough to still be distinguishable from spam. In our environment the filter can tell the difference between wanted and unwanted mails coming from the same opt-in vendor newsletter to multiple business recipients.

If you guys had a DFW office I might be convinced to come show you. ;-)

And self-updating classifiers are Russian roulette.

Is there ANY non-experimental filesystem that runs on a standard desktop OS that has this feature?

Waybackfs on
Linux.

Seagate Research presented a good technical article on SCSI vs. SATA back in 2003. Much of this is still relevant today (though it's SAS vs. SATA)

The fact that you're posting this as an AC and the impressive list of references really make me want to believe you. Maybe you've accidentally clicked the AC button while trying to paste a link to http://www.usenix.org/publications/library/proceed ings/sec96/full_papers/gutmann/
That paper talks about securely destroying data. There's a rather big difference between destroying every single bit on a hard disk in a way that it's impossible to recover, and causing a few tiny errors which force the user to run diagnostic tools all the time or send their disks to an expensive recovery center. Also, people will likely want to use their devices while charging them. Nothing is said about the effects of an oscillating magnetic field on read & write operations.

Whoa there. It is NOT bullshit. In fact it is COMPLETELY POSSIBLE to recover overwritten data from a hard drive, even if it was written over several times with random or nonrandom data. Remember that magnetic media cannot really store 1 and 0. It can only store a magnetic flux using ANALOG electronic components!

The NSA today (and other people) can use Magentic Force Microscopy to extract enough detail to reconstruct what used to be on the drive. With only one or two overwrites, a sensitive oscilloscope could suffice.

Here's one paper from ten years ago that talks more about the recovery technique.
http://www.usenix.org/publications/library/proceed ings/sec96/full_papers/gutmann/

From the paper:

"In conventional terms, when a one is written to disk the media records a one, and when a zero is written the media records a zero. However the actual effect is closer to obtaining a 0.95 when a zero is overwritten with a one, and a 1.05 when a one is overwritten with a one. Normal disk circuitry is set up so that both these values are read as ones, but using specialised circuitry it is possible to work out what previous "layers" contained. The recovery of at least one or two layers of overwritten data isn't too hard to perform by reading the signal from the analog head electronics with a high-quality digital sampling oscilloscope, downloading the sampled waveform to a PC, and analysing it in software to recover the previously recorded signal. What the software does is generate an "ideal" read signal and subtract it from what was actually read, leaving as the difference the remnant of the previous signal."

See Magnetic force microscopy (MFM)
http://www.usenix.org/publications/library/proceed ings/sec96/full_papers/gutmann/index.html

Nothing you erase is just magically "gone" unless you put *a lot* of effort into making sure it is gone. If the owner made sure that it was gone using a millitary grade wipe utility then statistically they should be able to prove that too! Even then they need proof that the owner was smart enough to be capable of performing this kind of disk wipe and a complete reinstall of the OS.