Slashdot Mirror
Why Not To Shout At Your Disk Array
Brendan Gregg of Sun's Fishworks lab has an interesting video demo up at YouTube demonstrating just how bad vibes, if expressed with sufficient volume in front of a rack full of disks, can cause a spike in disk latency. White noise, evidently, doesn't do them much harm. (Maybe they just feel awkward to get yelled at on camera.)
I always thought niggers' computers were slow because they didn't know how to use 'em but now I guess it has a lot more to do with the fact that niggers are loudly hootin' and hollerin' all the damn time.
I would never hire an overgrown chimpanzee to work with me at my Pixar rendering farm, that's for damn sure. Good CGI depends on Maya, not on mayates .
he's like the crocodile hunter of loud server rooms
Imagine a Beowulf cluster of those things!
No Sigs!
People yelling too much at their computers
It's been known for a long time vibrations are not good for discs (see notebooks). Even by early 90s music CDs had skip protection. If a disc skips, latency will of course momentarily increase. And with tolerances down even further, it's probably worse than back then.
In 10-15 years it won't matter anyway, almost everything will have SSD by then.
I wonder if the latency would vary by the pitch and tone of the person yelling. If that's the case, I'd wonder if that could be extrapolated into reconstructing whatever was being said. Granted, if you're yelling that loud, the person in the next county is more likely to hear you first.
It bothers me,
How this guy actually made the discovery.
He must have let off quite a bit of steam towards that rack.
...why?
I'm sure there's a good reason, but... He's using dtrace, right? Thus implying Solaris? Thus implying ZFS?
If you've got ZFS, why would you do JBOD?
Or did I just mis-hear him?
Don't thank God, thank a doctor!
Might the the drives themselves be sensing the induced vibration via an embedded accelerometer and momentarily parking the heads to avoid damage? It seems like the marketing folks shouldn't have too hard of a time putting a positive spin on this behavior.
if expressed with sufficient volume in front of a rack full of disks
I wonder about the results of eliminating the superfluous "full of disks" part.
Interestingly enough, it's precisely here, where the omission would both be understood and not bring unwanted connotations.
Now when Skynet finally goes sentient, it'll sue for emotional abuse. I thought metal death machines were bad, but now Lawyer-bots? We're doomed.
I doubt that white noise vs. voice has anything to do with it. He's yelling *right* in front of the disks - his voice is going to induce a lot more vibration just because he's so much closer than the equipment. Inverse-square decay and all that ;-)
Stud dogs go about the whole sex thing rather differently than primates (or equines). Unlike us, male canines don't have an orgasm that involves a short, intense ejaculation. Instead, once they have become fully erect, they will have a continuous orgasm for from 10 to 45 minutes or longer. The "standard" procedure for dogs, when they are mating, is that the male "ties" with the bitch - which means that, after he has penetrated fully, his penis will develop a knot at its base that is several times wider than the rest of his shaft.
For reference, a 80 pound Golden stud dog might have, let's say, a cock that is 7 or 8 inches long when erect - but his knot will be at least as big around as a tennis ball. This knot swells inside the bitch, and so long as he remains erect the dogs are "tied." No, this isn't painful for her - canine females long ago developed an entire set of muscular supports for this process. Generally, once they are tied, most stud dogs prefer to step off and over, so he and the bitch are tail-to-tail. Theories abound on why this evolved - I have yet to see one that was truly convincing. Anyway, they'll stand like this, with the male having a continuous orgasm during the whole tie - until he starts to shrink and they pop apart. Bitches also have orgasms, and she'll likely have quite a few during the tie, as well - research has shown that her orgasms are essential to increasing the chances of pregnancy, due to muscular contractions.
Anyway. if a guy like me has a stud dog partner, one form of intimacy is for him to tie with us, anally. As young teenagers, many of us learned the hard way about the knot, and the tie - particularly back in pre-interweb days. So we'd suddenly find ourselves locked together, with this tennis-ball width cock inside us. Nowadays, I suspect most young zoos know all about this. However, some folks still have eyes bigger than their stomach, err their you-know-what.
It would not be accurate to say that I have a stream of visitors who show up at my house just for sex with my canine partners. However, it is true that I do not exercise any sort of unilateral control/ownership over the relationships my canine boys might develop with other people - they are adults, and if they desire to get frisky with another two-legger and I judge that the person is respectful and unlikely to do anything mean or stupid, I have no moral ground on which to say "oh, no, you aren't allowed - he can only have sex with me." That just makes no sense, so if there's a time when a friend is visiting and there's a spark between them and one of my partners, I'm ok with that. In truth, I think it's great to have the boys' enjoy other positive relationships and I love to see them happy, whatever the circumstances.
Many years ago, a friend was visiting - a zoo who had been active with his own stud dog for quite a few years. His boy was a breed that is not small, but is also somewhat known by old-school zoos as being, well, on average not so well-endowed relative to their body size. This friend had tied with his partner on a number of occasions - and he often talked about how intense and rewarding the experience was, for both of them. That's great, I said - while thinking that he'd probably not fare so well with a larger breed.
As it turns out, he and one of my canine friends hit it off quite clearly right from the get-go - the chemistry was there and the two of them seemed like they'd known each other for ages. After several visits, I could see that they were sort of getting closer and closer - my friend was worried that I'd feel he was somehow intruding into my relationship with this handsome stud dog - who had been in my own family for close to a decade. Of course not, I told him - if you guys hit it off and things get steamy, I'd hardly throw cold water on it just so I can be all possessive and insecure. HOWEVER, I warned him, that handsome boy with whom you're making goo-goo eyes is much bigger than your own long-time partner.
I tried to be nice about this, but some zoos get the
Nice, now I can use this to detect if people were loud in my server-room.
Do they also react to smell?
I heard that there is a place where they throw chairs at things...
you need to play classical music to your disks, and they'll perform even better.
Well, partly at least. It's no secret that disk drives are sensitive to vibration as this video showed an extreme case. Keep in mind, since disk drives are spinning at 7200-15000RPM, they themselves create vibration that can affect adjacent drives. The drive enclosure can help reduce the problem with use of shock absorbers and vibration dampeners. Most drive enclosures nowadays, for cost reasons, are no more than just sheet metal wrapped around power supplies, fans and drives, which contribute to the problem.
hmm, bit of a chicken and the egg scenario there, isn't it?
is it slow because you yell at it, or do you yell at it because it is slow?
Either way, in the end it only degenerates into a downward spiral, where the computer gets slower and slower, while you get more and more pissed off at it and yell louder...
I am not stubborn. I am right!
Secret Fact : Ultrasonic noise at low volumes is WORSE !
It took weeks to testing to get to the root issue of WD Raptors dropping in head seeks on very high end raid cards in tiny head movement seek benchmarks, but padding each JBOD drive in acoustic foam (shooting range foam), or testing one drive at a time, instead of 4 or 8, (either method works) increased I/O per second by 40% in a rack chassis.
40% more head movements per second if no ultrasonic noise entering drives !!!!!
This is VERY VERY RARE INFO, and only I, the head of Gigabyte in Asia, and two engineers in california know of this discovery.
And because I know no one on Slashdot will mod this up, and no one reads at 0 anymore, I can trust my astounding well researched secret shall remain secret.
Its sadly 100% factual.
Or maybe it's even more complicated than that. Perhaps the user first shouts at his pc because he expects his machine to be crummy on windows and the first slightest mistake is magnified in a placebo effect.
That then begs the question, which came first: window's horrible reputation due to people yelling at their machines or people yelling at their machines due to window's horrible reputation?
I think that this discovery is just the beginning of a new and harmonious spiritual relationship between man and machine. Just imagine, DC engineers across the world singing beautiful harmonies to their storage arrays!!
Wonderful. ;-)
White noise is just like white light: An even distribution of energy across the spectrum.
It makes perfect sense that white noise is less of a problem than an equal amount of noise at a specific frequency. Given a suitable frequency the material absorbing the energy will vibrate and even resonate. (That particular engineer's yelling, apparently, resonates well with the disks in the array of that video...)
It's the same reason why a bullet-proof west can make the impact of a bullet non-lethal: Spread out distribution of energy.
.: Max Romantschuk
... would increase my latency too.
Having seen this, I wonder what effect the 'anti-vibration' rubber grommets that are used on most modern desktop PC hard drive bays have on disk latency. After all, they stop vibrations being transmitted into the case my allowing the HDD itself to vibrate more and damping the movement as it reaches the case. Of course, having the HDD vibrate of its own accord is much better than having it resonate with another component in the case, so perhaps in some cases, the damping is beneficial to latency aswell.
In fact, I'm surprised that no-one has come up with a case in which all parts have a natural frequency that does not coincide with the speeds of modern hard disks, such that the case will vibrate with HDDs, but not resonate, which is where most sound problems (the reason for damping in the first place) come from.
I was thinking ( more like dreaming ) about buying a HD video camera that uses a hard drive. I wonder how much the noise of flying in a WW2 era Warbird would mess with it. I flew in a B-25 and it someone screwed with my camera's ability to focus.
Taco?
Also from Brendan Gregg comes the always useful /usr/bin/maybe. Other funnies from him here.
I suppose percussive maintenance is likewise discouraged.
go and measure your own performance degradation while your hard disk does something mean to you
I dub this guy the disk whisperer ...
I'm glad Sun have got enough free time and energy to find out useless nonsense like this.
You thought you could break the laws of physics without paying the PRICE?
So, it's like a positive feedback loop ?
The only saving grace for optical media and hard drives will be their cost.
Optical media is damn cheap on a $/unit basis. Portable flash drives are never going to be under a dollar each. There's no profit to be made.
Now that this has been demonstrated, I'd like to ask for generous funding and unlimited access for doing studies to demonstrate the effects of good vibrations on hardware.
In case anyone hasn't seen it, I think I'm the first to have a data center blessed by a Tibetan Buddhist Lama. http://picasaweb.google.com/nagardner/200709LamaSamtenBlessesTheInternets#
I am currently hoping to see research on the long term effects of hanging prayer flags in data centers. My hypothesis is that if they would make for nice photos and surveillance videos, and would have a net improvement on the MTBF of hardware and personnel.
LOUD NOISES
That thing can shake my whole apartment. Is anyone testing what effect this thing has on my hard drives when listening to music/playing a game?
Years back I broke my ankle and had it screwed together for awhile. The screws are out now, but I've got them saved in a container and - other than the material - they really don't seem much different from those in my shop.
So was there some doctor who moonlighted as a carpenter, and one day looked at a broken, out-of-place bone... then at his workbench wood-projects ... then back at the bone.
Even today, looking at the X-ray of those screws firmly drilled through my bones gives me a bit of a creepy feeling, but I can't really complain about the results as the bones firmly knit themselves back together in the right positions.
Seems like I have a lot to worry about, I have a couple of disk arrays inside a 40-ft trailer that tends to rock in the wind...
The racks are shock-mounted, though.
you insensitive clod! Give them a bit to recoup!
I can stand brute force, but brute reason is quite unbearable. There is something unfair about its use. It is hitting be
I'm not being cynical in any way when I say I spent the last 1/2 hour checking out Fishworks due to the interesting analytics screen he used during the video. It occurred to me just now that's some (whether intentional or not) pretty effective viral marketing. LOL. The bad news is it appears Sun hasn't released Analytics as part of the base OpenSolaris distribution just yet. Too bad. I could just as easily use it to look at Veritas storage, assuming Veritas played nice with DTRACE. It'll be interesting to see what this looks like when Sun uses this on top of iSCSI served SAN disks instead of NAS. Don't know about you, but I don't park database filesystems on NAS. Perhaps that is outmoded / old-school thinking but that's the way it is.
...yelling at your hard drives makes them upset, then imagine how they'll react if you whip out your wong and start peeing on them.
Perhaps the converse is true? Are Macs more reliable because their owners are happy (and never yell)?
sounds right to me.
Disk drives have a resonant frequency
I've seen dramatic demonstrations of this over the years. One that stands out was a test of a Bryant drive sometime around 1970. In those days a 2 GB drive was at the edge of the envelope and Bryant was test-marketing just such a beast. It consisted of eight four-foot platters mounted four to a side on a shaft going through a monster of an electric motor. The heads were mounted on arms whose positioning was controlled by hydraulic cylinders big enough to be used as shocks on a pickup truck. The whole thing would not fit in the back of that pickup truck.
We were testing the thing with a program called the "Leese Bomb". Leese can identify himself or remain anonymous--I won't turn him in. The "Bomb" part was the nature of the test.
Basic tests in those days would involve writing a whole track and then reading it back and comparing what was read to what was written. You'd do this a number of times with different patterns to capture not only faults in the surface, but any sloppiness in the head control. The Leese bomb went one better.
It would write to the outside track, write to the inside track, read the outside track, read the inside track, and then compare. If the comparison failed it would repeat the test, and keep repeating untl it succeeded, counting the failures. If the test succeeded it would index the test both inward and outward so that the tracks tested would move toward the middle, cross, and continue. This test was superior in that it would capture dynamic flaws in the system as the distance the heads moved, and the time to move varied from max to zero.
In the case of the Bryant Drive (and, accidentally, an innocent Ramac drive at Caltech), the test found a resonant frequency. When the heads overshot their mark causing an error, the test stayed on the back and forth pattern, reinforcing the resonant motion with each cycle of the test. The drive started walking across the test floor in three-inch hops, but not for very long. In a few seconds, one of the shafts broke and one of the platters, a 500 pound disk rotating at 2400 rpm broke through the front of the unit and flew across the building until it was stopped, explosively, by one of the steel columns supporting the roof of the building. Miraculously, no one was hurt.
We gave up on Bryant for that application. Not long after that, CDC introduced its 200MB drives, and they passed the Leese Bomb with flying colours. Ten of them didn't take up any more room, or cost more, than the big Bryant, so our client was happy to go with that solution.
In any case the lesson is that, if it has moving parts, resonance is an issue.
I'm a Programmer. That's one level above Software Engineer and one level below Engineer.
In the wake of the Great Crash of 2008 that might take on scary new meaning.
My first Journal Entry ever, in 8 years! http://slashdot.org/journal/365947/aphelion-scifi-fantasy-horror-poetry-webzine
But is his name a killing word?
/came for a Dune reference, left disappointed
and I just remembered this kid maybe if he was a little bit calmer the game would have loaded ;)
I am not stubborn. I am right!
Does it matter? Because they are on drugs anyway. You could switch off the screen, and they still would see the pretty oozing colors of...ehem... "Photoshop"...
Any sufficiently advanced intelligence is indistinguishable from stupidity.
They found out a similar approach with toasters in Ghostbusters. Although, I don't think I saw his rack dance.