Whose Bug Is This Anyway?

← Back to Stories (view on slashdot.org)

Posted by Soulskill on Tuesday December 18, 2012 @01:04PM from the it's-nobody's-fault-and-everybody's-angry dept.

An anonymous reader writes "Patrick Wyatt, one of the developers behind the original Warcraft and StarCraft games, as well as Diablo and Guild Wars, has a post about some of the bug hunting he's done throughout his career. He covers familiar topics — crunch time leading to stupid mistakes and finding bugs in compilers rather than game code — and shares a story about finding a way to diagnose hardware failure for players of Guild Wars. Quoting: '[Mike O'Brien] wrote a module ("OsStress") which would allocate a block of memory, perform calculations in that memory block, and then compare the results of the calculation to a table of known answers. He encoded this stress-test into the main game loop so that the computer would perform this verification step about 30-50 times per second. On a properly functioning computer this stress test should never fail, but surprisingly we discovered that on about 1% of the computers being used to play Guild Wars it did fail! One percent might not sound like a big deal, but when one million gamers play the game on any given day that means 10,000 would have at least one crash bug. Our programming team could spend weeks researching the bugs for just one day at that rate!'"

14 of 241 comments (clear)

Min score:

Reason:

Sort:

The memory thing... by Loopy · 2012-12-18 13:15 · Score: 5, Informative

...is pretty much what those of us that build our own systems do anytime we upgrade components (RAM/CPU/MB) or experience unexplained errors. It's similar to running the Prime95 torture tests overnight, which also checks calculations in memory against known data sets for expected values.
Good stuff for those that don't already have a knack for QA.
1. Re:The memory thing... by DMUTPeregrine · 2012-12-18 13:47 · Score: 5, Informative
  
  Unless you're trying to overclock.
  Admittedly that's a small percentage of the populace, even among people who build their own systems.
  
  --
  Not a sentence!
2. Re:The memory thing... by Runaway1956 · 2012-12-18 14:24 · Score: 3, Informative
  
  " Either it's DOA or runs forever."
  Nonsense. I bought 8 gig of memory about 4 years ago, for an Opteron rig. That computer recently started having serious problems, with corrupted data and crashing. I looked at all the other components first, then finally ran memory tests. Memtest failed immediately. I removed three modules and ran memtest again, it failed immediately. Replaced with another module, memtest ran for awhile, then failed. The other two modules proved to be good, so I am now running that aging Opteron with 4 gig of memory.
  Yeah, yeah, yeah - I realize a single person's anecdotal evidence doesn't carry much weight. I wonder what the statistics are though? As AaronLS already pointed out, these tests seem to indicate that my situation isn't very unusual. Components age and wear out.
  
  --
  "Windows is like the faint smell of piss in a subway: it's there, and there's nothing you can do about it." - Charlie Br
3. Re:The memory thing... by Alwin+Henseler · 2012-12-18 14:39 · Score: 5, Informative
  
  The defect rate on hardware is so low you don't need to - buy your stuff from Newegg, assemble, and install. Either it's DOA or runs forever.
  
  Look up "bathtub curve" sometime. Even well-built, perfectly working gear is aging, aging usually translates into "reduced performance / reliability", and any electronic part will fail sometime. Possibly gradually. Especially the just-makes-it-past-warranty crap that's sold these days. And there may be instabilities / incompatibilities that only show under very specific conditions (like when a system is pushed really hard).
  That's ignoring things like ambient temperature variations, CPU coolers clogging with dust over the years, sporadic contact problems on connectors, or the odd cosmic ray that nukes a bit in RAM (yes that happens, too). A lot of things must come together to have (and keep) a reliable working computer, so a lot of things can go wrong and put an end to that.
4. Re:The memory thing... by scheme · 2012-12-18 14:52 · Score: 4, Informative
  
  Yeah, yeah, yeah - I realize a single person's anecdotal evidence doesn't carry much weight. I wonder what the statistics are though? As AaronLS already pointed out, these tests seem to indicate that my situation isn't very unusual. Components age and wear out.
  Check out "A study of DRAM failures in the field" from the supercomputing 2012 proceedings. They have some interesting stats based on 5 million DIMM days of operation.
  
  --
  "When you sit with a nice girl for two hours, it seems like two minutes. When you sit on a hot stove for two minutes, it
5. Re:The memory thing... by Sir_Sri · 2012-12-18 15:42 · Score: 3, Informative
  
  Even if you have a small calculation failure rate, it's not practical for an end user to recognize that as a hardware partial failure rather rather than a software bug.
  From the perspective of the average user, yes, it either works or it doesn't. If you use something bit (like wow/guildwars or the like) and they can diagnose it for you then you might have an argument. But even then, 1% could be overclocking or, as the author of TFA says, heat or PSU undersupply issues. That's not 'defective' hardware, that's temperamental hardware or the user doing it wrong. And because it's rare it's not necessarily serious, most users can handle the odd application crash in something like an MMO once every few days.
  It does mean a bug hunter needs to know what is happening though.
6. Re:The memory thing... by mcgrew · 2012-12-19 04:06 · Score: 3, Informative
  
  My experience goes along with this. A few times I've had dual-boot computers constantly crashing on the Windows side, so I was blaming MS for their buggy software -- until the flaky hardware that made Windows flaky failed completely. Turns out that Linux is simply far more hardware fault-tolerant than Windows, rather than Windows being a bug-ridden piece of shit.
  
  --
  Free Martian Whores!
7. Re:The memory thing... by Lonewolf666 · 2012-12-19 04:31 · Score: 4, Informative
  
  Intel also charges you extra for ECC (only in server processors and mainboards), while AMD supports it in their better desktop processors. You still have to check if the mainboard does support it, though.
  A quick online price check shows that for 8 GByte DDR3 RAM (2 sticks), you might have to pay 20 Euros more for the ECC variety, compared to non-ECC from the same vendor. The more limited choice in mainboards might end up costing you cost another 10-20 Euros, so let's say +40 Euros to get your AMD PC with ECC Ram.
  On the Intel side, it is more like +50 Euros for a small Xeon instead of a matching i5, +100 Euros for an ECC-capable board and the same +20 for the RAM as with AMD. That makes about +170 Euros to get an Intel with ECC RAM, and was the main reason why my current PC is still an AMD...
  
  --
  C - the footgun of programming languages
8. Re:The memory thing... by mgbastard · 2012-12-19 04:34 · Score: 3, Informative
  
  Without the paywall: The study was performed on the Jaguar Supercomputer at Oak Ridge National Laboratories http://softerrors.info/selse/images/selse_2012/Papers/selse2012_submission_4.pdf
  
  --
  Anyone seen my low uid? last seen 10 years ago while panning the #@$# out of Taco's 'web based discussion system'
OsStress by larry+bagina · 2012-12-18 13:20 · Score: 5, Informative

Microsoft found similar impossible bugs when overclocking was involved.

--
Do you even lift?
These aren't the 'roids you're looking for.
1. Re:OsStress by epine · 2012-12-18 21:48 · Score: 3, Informative
  
  Nope! It's the same processor. Sure, some come out different, but oftentimes there are loads of perfectly good processors that get underclocked for marketing reasons only.
  When the day arrives that we achieve molecular assembly, even then for two devices identically assembled with atom for atom correspondence, there will likely be enough variation in molecular or crystaline conformation remaining to classify the two devices at the margin as "not quite the same".
  Binning levels are determined by the weakest transistor out of billions, the one with a gate thickness three deviations below the mean, and a junction length a deviation above. There is probably some facility for defective block substitution at the level of on-chip SRAM (cache memory), and maybe you can laser out an entirely defective core or two.
  As production ramps, Intel has a rough model of how the binning will play out, but this is a constantly moving target. Meanwhile, marketting is making promises to the channel on prices and volumes at the various tiers. There's no sane way to do this without sometimes shifting chips down a grade from the highest level of validation in order to meet your promises at all levels despite ripples experienced in actual production.
  Intel is also concerned--for good reason--about dishonest remarking in the channel. There's huge profit in it, and it comes mainly at the expense of Intel's reputation. Multiplier locks help to discourage this kind of shady business practice. So yeah, a few chips do get locked into a speed grade less than the chip could feasibly achieve. This is all common sense from gizzard to gullet. What's your point, then?
  
  If they were an engineering firm, they'd sell one product at one price and be done with it.
  Where you even find so many stupid engineers? The College of Engineering for Engineers Who Think Statistics is One Big Cosmic Joke presided over by the Edwin J. Goodwin Chair of Defining Pi As Equal to 22/7?
Re:I don't believe 1% of computers give wrong answ by godrik · 2012-12-18 13:53 · Score: 4, Informative

I actually believe it. I am sure they might have think of floating point precision problem. But most likely they only used integers. That's what prime 95 and memtest are doing. Integer and memory operations uncover most common hardware failure. I encountered many computers with faulty hardware when stressed. And I am sure guildwars was stressful.
Re:Reminded me of my first C application by safetyinnumbers · 2012-12-18 15:15 · Score: 4, Informative

That's known as "Yoda style"
Re:Reminded me of my first C application by richardcavell · 2012-12-18 15:46 · Score: 5, Informative

I just want to correct this, not to prove how smart I am but because there are novice programmers out there who will learn from this case. The statement:

if (i = 1) {

is equivalent to:

i = 1; /* correction */ if (i) {