Slashdot Mirror


How Did Volkswagen Cheat Emissions Tests, and Who Authorized It?

Lucas123 writes: The method by which Volkswagen diesel cars were able to thwart emissions tests and spew up to 40X the nitrogen oxide levels set by the Environmental Protection Agency was relatively simple. It was more likely no more than a single line of code used to detect when an emissions test was being performed and place the emissions system in an alternate mode — something as simple as a software "on/off" switch. Volkswagen AG CEO Martin Winterkorn, who stepping down as the result of his company's scandal, has said he had no knowledge of the emissions cheat, but software dev/test audit trails are almost certain to pinpoint who embedded the code and who authorized it. You can actually see who asked the developer to write that code," said Nikhil Kaul, a product manager at test/dev software maker SmartBear Software. "Then if you go upstream you can see who that person's boss was...and see if testing happened...and, if testing didn't happen. So you can go from the bottom up to nail everyone."

20 of 618 comments (clear)

  1. Nail everyone? by Anonymous Coward · · Score: 5, Insightful

    Correction: "You can nail everyone that's in the official audit trail."

    The people at the top that authorized it (or at least didn't condemn it) probably never actually sent a traceable e-mail to anyone. Nor did they touch any code. Nor do they appear in any meeting minutes. These sorts of discussions tend to happen over a drink in a bar somewhere, and for good reason.

    1. Re:Nail everyone? by mbone · · Score: 4, Insightful

      Yeah. I would not be too surprised if at some level in the organization this was sold as a debugging or trouble shooting measure, or some other benign reason was given for branching on detection of emissions tests.

      In other words, the engineers who actually did the code may not have known the real purpose of what they were doing. Somebody knew, of course, but they may be harder to track down.

    2. Re:Nail everyone? by Anonymous Coward · · Score: 0, Insightful

      I don't know about German companies, but if you are a US company with a dev team of H-1Bs, you have a physical meeting, tell that you want some stuff... and they will do it. If they don't, they get deported, and once deported, they likely will never be able to come to the US again, since there are hundreds to thousands of workers hoping for each position.

      That is why you never read about a H-1B blowing the whistle on bad practice.

    3. Re:Nail everyone? by Archangel+Michael · · Score: 5, Insightful

      You get fired now, or you implement something dubious - what do you choose?

      You get fired. Then you sue for wrongful termination. Then you expose the company in court (public record) about how shitty they are threatening you with termination (you have proof? right?) for doing something dubious.

      One thing I have learned is you always say "Can I get that in writing?". This alone stops a huge number of stupid decisions, especially when you're protesting.

      Even if you have to write it ... "Per our conversation regarding _______ I am doing _______ at your request. Please let me know if you change your mind". In civil court, all you need is 50+%. It doesn't take much to get to 50+%. Self documentation is perfectly acceptable.

      It has saved my bacon a number of times.

      --
      Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
    4. Re:Nail everyone? by Anonymous Coward · · Score: 3, Insightful

      Yes and in the mean time you lose your house, car, and your kids go hungry. And the woman that would love you till death do you apart? She meant financial death. She will leave you for putting your family's financial livelihood in jeopardy to do some half-ass ethics play. You may get some money from the settlement, but the whole thing will now be part of public record. Good luck getting hired ANYWHERE else.

      It's easy to be an idealist on paper, especially if you have nothing to lose. Real life is a tad more complicated.

    5. Re:Nail everyone? by mbone · · Score: 5, Insightful

      Yes, but this might even have been sold as a compliance issue - we must by law make sure that the full emissions package is in place for any emissions test, even if the service tech turned it off for reason XYZ. What engineer would blink at that? Meanwhile, over in another department, the engineers are being told, these emissions packages must by default be off, as not all jurisdictions require them, and we are using an opt-in type system to turn them on when required. Again, who would blink at that? But by such stratagems, they could set up to that one person or a few people could flip a virtual switch, and the hack would be in place.

      Somebody knew, somebody high up knew, but I rather doubt that everyone on the engineering bench knew, and that means that they had to be fed plausible stories along the way.

    6. Re:Nail everyone? by Eunuchswear · · Score: 5, Insightful

      Oh, great. If you have it in writing you can say "I was just obeying orders". In a fucking German accent.

      That'll go down well.

      --
      Watch this Heartland Institute video
    7. Re:Nail everyone? by orgelspieler · · Score: 4, Insightful

      That's ridiculous. There are valid reasons for the code to have different operating modes. It is not inherently unethical to make software behave differently under different operating conditions.

      Think about it, VW sells cars in multiple jurisdictions. They need to have different emission mitigation regimes for each of them. It's possible that the flags for the operating conditions just got screwed up. If they use the same shop/lab model to pass EU and USA testing, there must be a switch in there for "Detected_testing_regime == EPA." And they might have intended to turn on the same controls when "vehicle_sold_to == USA," but the guy responsible for that code screwed something up.

      Even if it were an intentional deception, it would be trivial for management to divvy up the workload such that the individual programmers had no idea. "Jan, we need you to make a module that detects the following operating conditions and pass this parameter to the emissions controller, mm-kay?" or "Hans, when you see the disable_emission_control_for_emergency_use, make sure that the car switches to the high_power curve."

      We really don't know. Until the results of the investigation are made public, and they are found guilty of wrong-doing, they are innocent. Is the legal equivalent of "assume good faith."

  2. Aw... by 93+Escort+Wagon · · Score: 5, Insightful

    It's cute how he thinks no one thought about this and sanitized the audit trail. I'm sure he also thinks his 4096-bit disk encryption thwarts even the most determined ne'er-do-wells.

    --
    #DeleteChrome
  3. Single line of code? by rekoil · · Score: 4, Insightful

    I *highly* doubt it was a single line of code. To toggle the car's "EPA Cheat" mode, maybe, but by all accounts, the system used a variety of inputs to detect artificial driving conditions (including, apparently, barometer data), as well as needing code to define what engine parameters to change once the mode was entered.

    1. Re:Single line of code? by OverlordQ · · Score: 5, Insightful

      Or, "If they're cheating on this, what other things did they cheat on?"

      --
      Your hair look like poop, Bob! - Wanker.
    2. Re:Single line of code? by Chris+Mattern · · Score: 4, Insightful

      Yes, but people run these cars on dynos all the time for performance figures, and they all saw the standard performance figures. They fixed it so that the cheat mode would kick in when EPA testing was done on a dyno, and only when EPA testing was done on a dyno. It's doable, thanks to the EPA's rigid testing protocol, but it wouldn't have been simple.

  4. Keep good records of dubious orders from mgmt by sjbe · · Score: 4, Insightful

    I did get an email verifying that I had questioned it, but then I found out that all our emails are automatically deleted after 6 months or something like that.

    Nothing prevents you from printing emails of instructions to implement dubious decisions. I've done this from time to time just to protect myself when I worked at a large company.

    You get fired now, or you implement something dubious - what do you choose?

    If it is clearly illegal or will be very likely to cause major problems then you should seriously consider walking. If it isn't so clear then you get them to document their instructions to you and you keep a copy (print if you have to) for your records to cover your ass should it be a problem down the line. Make sure you document your objections and make it clear that you have taken every reasonable effort to ensure that what you are doing is legal. If the decision is merely dumb but legal, same thing but don't worry so much about ensuring legality.

  5. Professional Engineers have the power to say no by Joe_Dragon · · Score: 4, Insightful

    Professional Engineers have the power to say no and they have Ethics rules to fall back on.

  6. The engineers knew what was happening by sjbe · · Score: 4, Insightful

    Somebody knew, somebody high up knew, but I rather doubt that everyone on the engineering bench knew, and that means that they had to be fed plausible stories along the way.

    Spare me. The engineers were the ones that eventually spilled the beans on this. They weren't fooled by some clever management strategy. They knew exactly what they were doing and they knew or should have known that it was illegal. While maybe not every engineer involved knew, more than a few certainly did without question. The engineers at VW aren't dumb. I know a few personally. Please stop with the attempts to find clever ways to not have to blame the engineers who were guilty of helping to commit fraud. Management may have ordered the crime but the engineers were the ones that carried it out.

    1. Re:The engineers knew what was happening by Calydor · · Score: 3, Insightful

      Is it not possible that years after these things were made by two different development groups as the GP suggested, these two groups happened to discuss the work they had done, blinked, and realized how those two things worked together. Or given this started back in 2009, there's probably been some turnover, no one in charge knew anymore this was happening and had to be kept secret, and one person was given full access to the code to add a few new snippets - and HE realized what it was actually doing.

      --
      -=This sig has nothing to do with my comment. Move along now=-
  7. Re: Professional Engineers have the power to say n by afeeney · · Score: 5, Insightful

    It's fascinating to see how many posters here automatically assume that it must be the PHBs who pressured the engineers into this. Very few assume that the engineers saw an opportunity for a bonus or for the PHB to owe them one, and added the cheat function voluntarily. I've not seen any posts so far that suggest an engineer thought of the cheat and suggested it to a PHB.

    A reminder that we tend to think of our peers as being much more ethical than "them" and look for reasons to think of them as victims of force or circumstances, and assume that "they" are only motivated by sheer callous greed. Whoever the "them" is.

  8. Re: Professional Engineers have the power to say n by AK+Marc · · Score: 4, Insightful

    And I find it hilarious that everyone here states that programmers and engineers work without requirements or documentation. I've worked places where the verbal meeting would have the engineer agreeing with everything, then when it's not written in the requirements document and signed off by 10+ people, it doesn't get built. Seems like all the programmers on Slashdot have never worked in a company larger than 10 people.

  9. Re: Professional Engineers have the power to say n by Grishnakh · · Score: 5, Insightful

    It's an American thing.

    Employers here don't care about holding onto skilled programmers or other skilled people, because PHBs think they can just hire replacements on a whim.

    Yes, in reality new ones are hard to find and take a while to get up to speed. The PHBs will even acknowledge this when they're trying to hire.

    But once they have one employed, they don't care about keeping him happy, because they think they're al interchangeable cogs.

    If you're seeing a giant disconnect here, yes, there is. This is how American corporations think; it makes no sense at all. I can't explain it. It's the same phenomenon where corporations will give a big salary offer to a new engineer, but once he's employed there, they'll just freeze his salary or give him paltry CoL raises, while giving new hires even bigger salaries, causing employees to switch jobs every 2-4 years (in Silicon Valley, it's 12-18 months).

  10. It is highly possible that "no one" at all did it by goombah99 · · Score: 3, Insightful

    While it is likely this was a sin of commission it remains plausible that no one did this at all. My thinking is that if instead of being programmed explicity the computer program was allowed to train itself for it's emission and performance tuning that a very natural outcome would be for it to learn to minimize emissions during emission type testing. Then on the test track it would learn performance and handling. etc... and so you end up with something that cheats but no one told it to nor was anyone even trying to make it cheat. It's just the result of getting what you optimize for.

    One reason that I like that theory is that if you consider the opposite, that it was a conspiracy, then this is not the sort of thing you can keep secret easily. You might succeed but that's pretty hard especially considering the time span and the inevitable entry of new personnel and suppliers into the supply chain. So I don't think this was intentional. The exception might be if if it's a conspiracy of one. that for some reason there was just one guy who could pull off everything. THen you would have a shot of keeping this secret.

    --
    Some drink at the fountain of knowledge. Others just gargle.