Slashdot Mirror
Elevation Plays a Role In Memory Error Rates
alphadogg writes "With memory, as with real estate, location matters. A group of researchers from AMD and the Department of Energy's Los Alamos National Laboratory have found that the altitude at which SRAM resides can influence how many random errors the memory produces. In a field study of two high-performance computers, the researchers found that L2 and L3 caches had more transient errors on the supercomputer located at a higher altitude, compared with the one closer to sea level. They attributed the disparity largely to lower air pressure and higher cosmic ray-induced neutron strikes. Strangely, higher elevation even led to more errors within a rack of servers, the researchers found. Their tests showed that memory modules on the top of a server rack had 20 percent more transient errors than those closer to the bottom of the rack. However, it's not clear what causes this smaller-scale effect."
Top of the rack tends to get toasty, but is this too simple?
Someone tell Fusion.io. They're based at 5000+ feet here in the Salt Lake valley! It would be interesting if their QC procedures are what have made them more reliable as the failure rate is higher where the testing is performed.
Do computers have a fear of heights?
Another reason for nerds to stay in the basement
This isn't news. Companies that make supercomputers have known this for decades. The one I worked for 15 years ago used a high elevation test environment in Colorado to verify error correcting capabilities. Even the article says that the results were not a surprise.
This message is encrypted with Quad ROT-13 to protect the author's copyright under the DMCA.
A couple of years back at one of the Supercomputing conferences (I think in Phoenix), Fermilab had a cloud chamber in their booth, and you simply *would* *not* believe the amount of ambient radiation passing you at all times. I can easily believe that altitude would have an effect.
Another interesting idea would be to do the same experiment by latitude. Does the Arctic Region Supercomputing Center have a higher rate than the Maui Supercomputing Center? What happens during an aurora?
Yeah it's gotta me shielding. Put two racks side by side with a lead or some other kind of shield from "cosmic ray-induced neutron strikes" on one rack. I'm guessing that gravity probably has a negligible effect on this entire test. it's probably more about the amount of atmosphere above the racks at different altitudes. -G
If you get high you can lose your memory?
According to the article the low elevation system was a Jaguar supercomputer whereas the high elevation one a Cielo supercomputer. Based on available specs for each the two are entirely different systems. How can they reach conclusions about altitude-relative bit error rates when they're not even comparing the system system? The article goes on to state:
"The group had found that, when all other possible confounding issues were factored out, Cielo's SRAM had a "significantly higher rate of SRAM faults," compared with Jaguar's SRAM, Sridharan said."
Huh? They factored out all confounding issues except that they were completely different systems.
It seems to me that an unexploited structure for a low radiation environment is the bottom side of a water tower. Steel has most radionuclides slagged off when it is produced while drinking water standards ensure the water in the tower will have low radioactivity. A meter or two of water forms a nice shield for cosmic rays from above while the air below the tower shields against lower energy ground radiation. And, you get a nice heat sink in the water for cooling electronic.
As one of the authors from the LANL-side I want to be clear that Sandia National Laboratories played a vital and at least equal role in this work - paper, analysis, as well as procurement and running of the Cielo supercomputer studied. The partnership with AMD, SNL, and LANL has been outstanding.
As this is not (mainly) about the system RAM, it's about the CPU caches, I wonder if any attempt is being made to correct the errors, and if it's worthwhile. One would just need to reset the node on any sign of an error, so the capactiy penalty would be small compared to ECC. On the other hand, the errors could just as well happen in the actual logical units, and at some point it's impossible or very expensive to protect everything. Because the SRAM takes up a large fraction of the CPU area, it may be useful to add something to protect the caches.
For some workloads you can do consistency checks in software, but for many computations that would require you to run the computations twice -- which is very expensive. Maybe statistical methods can be used to include a term for gross numerical errors -- different from the small floating point errors. It would probably be close to impossible to model the effect of such errors on the results though. Another option is to shield the datacentres from cosmic rays, if those are indeed the culprit.
Is this why when I'm in an airplane I can never remember if I turned all the lights out? ;-)
Lost at C:>. Found at C.
On Mount Everest, time slows by 0.00261261 seconds (2.6ms) compared to sea level.
Every foot higher you go is 90 billionths of a second difference, if you want to check the maths for me. The problem is, we're not talking about a sea-level / Mount Everest communication here. The RAM chips are about a foot long at absolute maximum.
And these sorts of effects then suddenly skitter into insignificance compared to solar radiation, different pressures, different air make-ups, heat, etc.
The fact is, we know that this effect exists. We know that time-slowing exists (GPS wouldn't work if we didn't compensate for such things). We know that solar radiation exists. But this single statistic barely bothers to eliminate memory manufacturer, operating voltage, or ambient temperature as a cause rather than these exotic causes.
Chances are, they might just have had a batch of dodgy RAM chips from a single manufacturer more than ANYTHING else combined.
And, even then, you'd need thousands of test sites / machines to even hint at the cause. But, why bother? We know there would be an effect, we also know it wouldn't be this large or obvious and that - chances are - there's a much simpler explanation. The whole "top of the rack fails more often" hints at what complete and utter bullshit this is. That would be an effect we'd notice at sea-level and most likely things like ventilation and heating have orders-of-magnitutide more to do with it.
Proves all my theories about tall people.
The time dilation between two places separated by the height h is approximately gh/c^2. With g~10 m/s^2, h~2000m and c^2~10^17 m^2/s^2, you get a time dilation of ~2*10^-13. In other words, about seven microseconds per century. That certainly cannot explain the different error rate.
. . . .recall that the new NSA "Supercenter" in Utah is at ~4300 feet. So they'll be making a lot MORE errors when monitoring us all. . .
The 1U lead block. Place at top of rack to protect the servers below.
Does it work? Who cares. If people will pay £150 for a wooden volume knob on their audio system, someone is going to pay whatever you ask for a lump-o-lead that may or may not improve the reliability of equipment below.
Why did anyone need to do this field survey? It simply confirms what we already know - cosmic rays create SRAM errors. Hot components fail more than cold components. Big whoop.
http://hyperphysics.phy-astr.gsu.edu/hbase/astro/cosmic.html
Earth act as a shield that protect memory from radiation coming from the other side of the planet. In addition, the collision probability of a particle is proportional to the distance of his travel into the atmosphere, so there is more probability on the ground to be hit by particle coming from the vertical. On a desktop computer the RAM is usually oriented vertically and exposing his shorter side from the top: the exposed area is very small for radiation coming from the top. Not that because of the motherboard orientation, this is also true for a lot of component mounted into it. On a server, the RAM might be vertical, but expose his largest side from the top. RAM mounted with a angle are even worse. Not that on server the motherboard is oriented horizontally, exposing most of the components with there biggest area from the top. So it's not a big surprise that the caches inside the CPU is hit more often on a server than in a desktop computer.
Write a cod that reserve several Go of memory on a non ECC memory, set it, and in a infinite loop check that all bits are set. Now try approach various type of arc lamps to the memory and count the number of hours (or minutes if you have a strong source) before a bit is detected to be zero. Now retry the experience with different RAM orientation.
There are statistics that cover the expected frequency of events caused by radiation in the first couple of pages.
http://docs.oracle.com/cd/E19095-01/sf3800.srvr/816-5053-10/816-5053-10.pdf
Either give it away or get top dollar, but never sell yourself cheap.
I'm also more prone to errors when I'm high
More music, fewer hits
From deep within the PDF (second link):
(Living in Colorado, I thought perhaps chips suffered from the same spurting newly opened toothpaste tube problem when manufactured at low altitude and installed into operation at high altitude, but it turned out the hypothesis was different, and, of course, left out of the Slashdot summary.)
What kind of materials (if any) are effective in blocking cosmic rays? Would it be possible to integrate cosmic radiation shielding into an average-sized PC case? If that's impractical, are there building materials that can be used in roofs and/or walls to block this stuff without breaking the bank?
But could it be simply gravity?
You mean because the 1 bits are lighter than the 0 bits? But you've got to remember about packing density. You can fit a lot more 1s than 0s because they are thinner. Vibrations in the chips will help the 1s settle to the bottom, despite being lighter.
When our name is on the back of your car, we're behind you all the way!
Then wouldn't you expect a cascading rate of failures from 20% down to the baseline bottom rack in a linear fashion?
The majority of cosmic rays that make it this far are muons. These are relatively penetrating and I highly doubt that a few centimetres of metal and plastic will have anything like a 20% effect. 60m underground with the ATLAS detector at the LHC we still get a reasonable rate of cosmic rays and we use them for calibration when there is no beam. While the rate is reduced 60m of rock is far, far more shielding than a few computers plus many cosmics passing through you come at an angle so the stack above will have no effect on shielding these.
I expect that heat and vibration will be the most likely causes.
You would need many metres to have a noticeable effect on the penetrating muons which make up the majority of cosmics at the surface. This should tell you that a few computer boxes is not really likely to have much of a shielding effect. This is reinforced by the fact that many cosmics come at shallow angles so the stack above provides no shielding. I doubt this is a cosmic ray effect.
It has more to do with the size of the circuits today compared to yesteryear. Smaller circuits are more vulnerable.
When our name is on the back of your car, we're behind you all the way!
"However, it's not clear what causes this smaller-scale effect."
Servers are made out of metal and have EM fields! This isn't hard.
Enough with all the mixing of terminology.
You use 'altitude' when referring to how high something is above the ground. You use 'elevation' when referring to how high the ground is from sea level.
What you don't see are signs for city limits on the road with 'altitude' on them. They say 'elevation' for a reason. Just like you don't find an elevatometer inside an airplane. You find an altimeter.
Mixing these terms as you've done (and so has TFS, so I don't blame you as you were simply restating the flawed summary) only causes confusion.
Someone flopped a steamer in the gene pool.
who's with me?
In aggregate entire atmosphere down to sea level works out to something like the equivalent of 30ft of water of shielding.. 20% reduction thru an entire rack of servers sounds to be in about the right ballpark.
People have been running the same experiments on international flights on laptops for years.
I have no idea if that's enough shielding to matter. However, if true, would we also see higher error rates in daytime when the body of the Earth isn't standing between the server and the Sun?
Cyrano de Maniac
An altimeter in an airplane is normally adjusted to show altitude above sea level (although above a certain height, it's set to assume a standard sea level pressure rather than what the current weather is producing). The two terms have very similar meanings, although I think elevation is more often used to refer to a fixed location.
Just curious. I've seen direct driven blowers in a number of various applications. Is there some special need to use belt driven blowers for the air in data centers?
How long before the cloud computing and storage services start charging a slight premium to have your stuff run/store on lower spots in their server racks?
SLASHDOT: news for people who can't concentrate on work or have no life at all and got tired of yelling back at the TV.
I know I have trouble remembering when I'm high. Seems like electronics should have the same problem.
While I guess why more ionizing events stemming from neutron impacts affects electronics, I don't get the blaming of "pressure"? Perhaps they mean the reduced air cooling of electrical components from a less dense atmosphere? Someone else noted that components on the top of a rack might tend to be warmer. This might be more of the same sort of effect.
Isn't it obvious?
About five years ago, I was involved in the installation of a thousand-node cluster in Boulder. We knew *before we went in* that we needed to change our EDAC (memory error correction) code to account for the higher rate of bit-flips due to the altitude. Some of the people we were working with had been there when those same problems nearly caused a months-long delay in a larger installation at NCAR nearby. We ended up running into a more subtle problem involving lower air density, heat and voltage, but *this* problem was incredibly old news even then.
Slashdot - News for Herds. Stuff that Splatters.
If cosmic rays were the cause (presumably from the sun since that's the closest source) then I would assume that latitude would be an important factor as well. Less sun means less cosmic radiation. My money is on the simplest explanation: Heat.
Time flies when you don't know what you're doing
Just like humans, computers have trouble remembering things when high.
On Mount Everest, time slows by 0.00261261 seconds (2.6ms) compared to sea level.
Error (in cmp_phenom_numer), missing argument 'scale'
That statement is void of any sense. The units of time dilation are not units of time. More like unitless (percent). Do you mean that a clock on the top of Everest is slower by 0.00261261 seconds per second, 0.00261261 seconds per year, 0.00261261 seconds since the dawn of time? What exactly do you really mean?
I doubt VERY much that the effect is 0.00261261 seconds per second (0.26%).
Honest, Mr NSA Sir, I was just searching for happy kitty cats and not "How to load a nuclear bomb in a suitcase" -- it must have been those nasty cosmic rays changing up my searches!
(I WONDERED why I kept finding lead suitcases with "Hello, Kitty" emblazoned on them.)
If the universe is someone's simulation -- does that mean the stars are just stuck pixels?
Almost no cosmic rays come from our local sun, mostly just slower solar wind particles.
"Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
Perhaps the researchers are too young to have read this 1979 paper http://www.ncbi.nlm.nih.gov/pubmed/17820742
I make new friends every day.
depends on her mood lol
Check out this Actel whitepaper (PDF). Describes a similar phenomenon, with such errors taking place three times more often in mile-high Denver than Baghdad by the Bay San Francisco.