Slashdot Mirror


How Did Volkswagen Cheat Emissions Tests, and Who Authorized It?

Lucas123 writes: The method by which Volkswagen diesel cars were able to thwart emissions tests and spew up to 40X the nitrogen oxide levels set by the Environmental Protection Agency was relatively simple. It was more likely no more than a single line of code used to detect when an emissions test was being performed and place the emissions system in an alternate mode — something as simple as a software "on/off" switch. Volkswagen AG CEO Martin Winterkorn, who stepping down as the result of his company's scandal, has said he had no knowledge of the emissions cheat, but software dev/test audit trails are almost certain to pinpoint who embedded the code and who authorized it. You can actually see who asked the developer to write that code," said Nikhil Kaul, a product manager at test/dev software maker SmartBear Software. "Then if you go upstream you can see who that person's boss was...and see if testing happened...and, if testing didn't happen. So you can go from the bottom up to nail everyone."

24 of 618 comments (clear)

  1. Nail everyone? by Anonymous Coward · · Score: 5, Insightful

    Correction: "You can nail everyone that's in the official audit trail."

    The people at the top that authorized it (or at least didn't condemn it) probably never actually sent a traceable e-mail to anyone. Nor did they touch any code. Nor do they appear in any meeting minutes. These sorts of discussions tend to happen over a drink in a bar somewhere, and for good reason.

    1. Re:Nail everyone? by PaulRivers10 · · Score: 5, Interesting

      Yeah, I worked for a large bank. I was told my task was to implement tax calculation code in javascript, so it would update on the page immediately. While I balked at the request, it was made pretty clear that either I do it or I would get fired. I was not given a javascript library that knew how to handle financial values. Javascript doesn't support integer-only values, so you're doing financial calculations with floating point roundoffs and errors. I happened to know about them but was not given any instructions that they were a problem. I wasn't given any instructions on how to make sure they didn't cause issues. As far as I know no more than basic testing was done on the code. I did get an email verifying that I had questioned it, but then I found out that all our emails are automatically deleted after 6 months or something like that. You get fired now, or you implement something dubious - what do you choose?

    2. Re:Nail everyone? by Archangel+Michael · · Score: 5, Insightful

      You get fired now, or you implement something dubious - what do you choose?

      You get fired. Then you sue for wrongful termination. Then you expose the company in court (public record) about how shitty they are threatening you with termination (you have proof? right?) for doing something dubious.

      One thing I have learned is you always say "Can I get that in writing?". This alone stops a huge number of stupid decisions, especially when you're protesting.

      Even if you have to write it ... "Per our conversation regarding _______ I am doing _______ at your request. Please let me know if you change your mind". In civil court, all you need is 50+%. It doesn't take much to get to 50+%. Self documentation is perfectly acceptable.

      It has saved my bacon a number of times.

      --
      Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
    3. Re:Nail everyone? by pla · · Score: 5, Informative

      The hard part here comes from "get it in writing".

      When someone three layers of food-chain above you tells you "do this", you don't get to refuse until you have it in writing (unless you already have a new job lined up - and even then, don't expect that one to go any differently).

      Now, you can certainly try to get them on record - You can ask them to write up a quick spec for what they want; you can ask them to submit the Change Management request because you don't have the authority to approve this one; you can send emails asking for clarification; and as a last resort, you can just document the change as "at the request of Boss X". In the real world, however, we've all dealt with people who refuse to do anything except by phone or in person.

      And at that point, it becomes your word against theirs. Guess who can afford the better lawyer? And even that assumes it completely blows up - If it remains an internal matter, you won't even get the chance to present your side of the situation, just pack your belongings up and GTFO.

    4. Re:Nail everyone? by mbone · · Score: 5, Insightful

      Yes, but this might even have been sold as a compliance issue - we must by law make sure that the full emissions package is in place for any emissions test, even if the service tech turned it off for reason XYZ. What engineer would blink at that? Meanwhile, over in another department, the engineers are being told, these emissions packages must by default be off, as not all jurisdictions require them, and we are using an opt-in type system to turn them on when required. Again, who would blink at that? But by such stratagems, they could set up to that one person or a few people could flip a virtual switch, and the hack would be in place.

      Somebody knew, somebody high up knew, but I rather doubt that everyone on the engineering bench knew, and that means that they had to be fed plausible stories along the way.

    5. Re:Nail everyone? by Agent0013 · · Score: 5, Interesting

      From the article itself, (I know, I am not supposed to read the article), the traction control is mandated to be turned off when testing the emissions. So you already have a need to check that emissions testing is being done. Once that flag is set in the code other areas can use it and they may not even be aware of the ethical dilemma that arises. Perhaps someone was told to lower the fuel flow when the test_mode flag was on. Why would they feel that something does not "smell quite right" in that case? Someone else turned the flag on with a code change several years previously. There is plenty of other things being changed or turned off when in test_mode, so adjusting the fuel flow can fit right in with that. Unless they realized that test_mode was used to determine that emissions testing was being performed (it could be used for other tests also) and that the changes they are making will cheat the emissions test they might not even realize that something is wrong with what they are doing.

      --

      -- ssoorrrryy,, dduupplleexx sswwiittcchh oonn.. -Quote found on actual fortune cookie.
    6. Re:Nail everyone? by Eunuchswear · · Score: 5, Insightful

      Oh, great. If you have it in writing you can say "I was just obeying orders". In a fucking German accent.

      That'll go down well.

      --
      Watch this Heartland Institute video
    7. Re:Nail everyone? by Wrath0fb0b · · Score: 5, Interesting

      Yeah. I would not be too surprised if at some level in the organization this was sold as a debugging or trouble shooting measure, or some other benign reason was given for branching on detection of emissions tests.

      Folks, you have to branch on emissions (and dyno) tests in the ECU solely because otherwise the safety side of things will bring everything to a halt. The most obvious reason (assume FWD) is that if the front wheels are going 65MPH and the rear wheels are going 0MPH, the traction control system is going to have a major freak out and say "HOLY CRAP WE ARE SKIDDING OUT OF CONTROL BRAKE FRONT UNTIL THE DIFFERENCE IS LESSENED". The procedure to enter dyno mode is not itself a secret.

      So it's not a secret and it's not illegal for the ECU to detect and behave differently during the test, and everyone would know about this above-board feature. The secret-and-illegal part is modifying the behavior of things measured by the test while the test is running. That is a lot easier to keep secret and requires a lot less involvement from teams directly outside the module that is responsible for emissions.

      [ Source: Tuner people that dyno fancy cars and have to solve these sort of issues. ]

    8. Re:Nail everyone? by Anonymous Coward · · Score: 5, Interesting

      IANAL, but this is what I heard from an employment lawyer (paraphrasing): As long as you wrote and sent the e-mail, the onus is on the employer to show that they have replied to it in a reasonable fashion. Courts tend to throw out "I didn't see it" excuses and expect replies. And even if they replied verbally, oh well, too bad, they should've done so in writing.

  2. Aw... by 93+Escort+Wagon · · Score: 5, Insightful

    It's cute how he thinks no one thought about this and sanitized the audit trail. I'm sure he also thinks his 4096-bit disk encryption thwarts even the most determined ne'er-do-wells.

    --
    #DeleteChrome
    1. Re:Aw... by Anonymous Coward · · Score: 5, Interesting

      Why would anyone want to hide evidence? All the people seem to operate under the impression that this is some kind of big revelation, when in fact everyone in the industry knows that this is happening all over the place. This wasn't actually big news that Volkswagen is cheating, the big news is that some authority is finally taking the rules seriously, when for years both the european and US authorities did their best to look the other way when the evidence was shoved in their face. And if no one cares, why would you want to hide it?

    2. Re: Aw... by Anonymous Coward · · Score: 5, Informative

      Definitely been going on for at least 17 years, and many other companies besides VW are getting caught.

      "On October 22, 1998, the Department of Justice and the Environmental Protection Agency announced an $83.4 million total penalty against diesel manufacturers, the largest civil penalty ever for violation of environmental law...The seven companies sold 1.3 million heavy duty diesel engines containing illegal "defeat devices," which allow an engine to pass the EPA emissions test, but then turn off emission controls during highway driving. As a result, these engines emit up to three times the current level for NOx a harmful air pollutant."

      http://www2.epa.gov/enforcement/detroit-diesel-corporation-diesel-engine-settlement

      You can find many more examples of enforcement at:

      http://www2.epa.gov/enforcement/clean-air-act-vehicle-and-engine-enforcement-case-resolutions

      (AC because I work in the industry.)

  3. Re:Single line of code? by ZeroPly · · Score: 5, Funny

    On the other hand, the code could be in Java. Those programmers are so verbose, all you have to do is search for the cheatOnEmissionsWhileRunningEPATest() functions.

    --
    Support microSD: in a post 9/11 world, it is unwise to carry your data on media that you cannot comfortably swallow.
  4. Re:Single line of code? by jandrese · · Score: 5, Interesting

    Supposedly the "cheat mode" is an extension of the "testing mode", where the car knows it is running on a Dyno because one set of wheels is turning at a high RPM and the other set are stationary. For a car with traction control this is normally a freakout event so they have to check for it and make sure not to go crazy just because the machine is strapped into a test harness. Once you have the otherwise required detection code in there, adding a single line to fully open the EGR valve when in that mode would be a piece of cake.

    --

    I read the internet for the articles.
  5. SubjectsInCommentsAreStupid by lesincompetent · · Score: 5, Interesting

    Someone should have leaked this a looong time ago. Perhaps some dev, why not.
    Hell it would have saved VW a lot of money! Think about recalling 1mln cars instead of 11mln!
    Did VW really think it could get away with this indefinitely?
    Fucking corporate morons...

  6. Re:Single line of code? by Anonymous Coward · · Score: 5, Interesting

    Sometimes but not always. This cheated the test both with and without a computer. It was instead detecting when it was on a rolling road. Emissions tests are always done with the car stationary but the wheels moving, and that'll be what the software was detecting.

    From the sound it of it wasn't actually putting the car into a special 'mode'. It was turning on all the measures to reduce emissions. When the car was on a real road it was turning them off to get better performance and fuel consumption at the expense of emissions. So it sounds like the car does technically meet the regulations, but ignores them when it's on the road. So expect the recall to turn them on at all times, which'll mean you don't need the car/engine replaced, but will mean you pay more at the pump and see your car's less nippy than it was before.

  7. Single line of code? I want to see this code. by KatchooNJ · · Score: 5, Funny

    EPA Cheat Code: Up, Up, Down, Down, Left, Right, Left, Right, B, A

    --
    "Never give up, for that is just the time and place when the tide will change." -Harriet Beecher Stowe ^_^
  8. How test mode was triggered by sjbe · · Score: 5, Informative

    It's been a while since I watched my car being tested but do they hook up the car to a computer terminal of some sort? Could those be used to trigger test mode?

    The test mode was triggered by monitoring which wheels were turning, position of the steering wheel, etc.

    Basically they wanted to avoid the cost of installing a urea injection system so they cheated instead. Honda engineers were reported to be perplexed about how they managed to do this miraculous feat of engineering.

    Here's a good article about what is known so far:

    http://www.msn.com/en-us/autos...

  9. Re:Single line of code? by OverlordQ · · Score: 5, Insightful

    Or, "If they're cheating on this, what other things did they cheat on?"

    --
    Your hair look like poop, Bob! - Wanker.
  10. Re:Single line of code? by AmiMoJo · · Score: 5, Informative

    According to news reports (BBC etc.) it used a combination of inputs. Steering wheel position, barometric pressure variation over time, rate of acceleration, speed, g sensor stability etc.

    Also, to enable the cheat mode the engine would have had to load a different set of operating parameters. Those parameters must be stored somewhere, and doubtless constitute more than a single line of code.

    Some thought must have gone into this fraud.

    --
    const int one = 65536; (Silvermoon, Texture.cs)
    SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
  11. Re:Sometimes the ethical path is very clear by hattig · · Score: 5, Interesting

    The engineers who designed the engine - yes.

    The engineers who coded the software for the engine computer? Why would they know what this does? The software is enabling a signal, hell the signal might even have a vague name, when a condition is met. The condition's name might not very clear.

    Yes, a spec saying "when the car is undergoing a test then enabled the cheat mode to get past the test" would clearly implicate the developers.

    But most likely it was: "when sig_x and sig_y and indicator_a are set, then set sig_Z to 1 in pattern P for n nanoseconds blah blah blah"

    Someone knew what they were doing, and it probably went like this:

    Engineer: We can't make this engine pass NOx tests.
    Message goes up the chain to a certain decision making level, possibly the board. Marketing chimes in: We can't have this, we're already sending out teasers about our new urea-less engine technology, etc, etc.
    Eventually a message comes down to fix it, in vague terms, entirely forgetting the original message that it's unfixable.
    Engineers: struggle for ages.
    In pub: Well, we could enable a special testing mode to pass the tests?
    In work: Shall we do this -> up the chain. Original context is half forgotten. Approved.
    Changes made. Software specs made. Timebomb implemented.

  12. It may not be a single line of code. by funwithBSD · · Score: 5, Interesting

    I worked for a Small software house that made SAP type ERP software before SAP ate the majority of the market. This was 1998 or so...

    We had a customer come to us and ask for certain modifications. Then a few more. Then a few more.

    Not unusual, we made a lot of money from change orders. So the first few were done. All were acceptable in the Generally Accepted Accounting Practices guidelines.

    Somewhere along the line the GAAP accountant realized that this last modification set would, taken in combination with all the other mods, make a check disappear from the system and become untraceable.
    We refused to do it, and the customer dropped the product, saying we were too hard to deal with. A million+ of revenue were lost, no small amount for the company.

    That customer? MCI Worldcom.

    They clearly had picked apart the source code and found the edge case that triggered the behavior. I had left the company before MCI blew up, but my understanding is that they were called to give testimony/evidence in the trial.

    This could be the same thing, a series of unrelated changes that trigger a diagnostic mode when hooked up to the test equipment.
    If so, it would be very hard to trace who made the ultimate decision to do this, as it might be spread across many teams working independently.

    --
    Never answer an anonymous letter. - Yogi Berra
  13. Re: Professional Engineers have the power to say n by afeeney · · Score: 5, Insightful

    It's fascinating to see how many posters here automatically assume that it must be the PHBs who pressured the engineers into this. Very few assume that the engineers saw an opportunity for a bonus or for the PHB to owe them one, and added the cheat function voluntarily. I've not seen any posts so far that suggest an engineer thought of the cheat and suggested it to a PHB.

    A reminder that we tend to think of our peers as being much more ethical than "them" and look for reasons to think of them as victims of force or circumstances, and assume that "they" are only motivated by sheer callous greed. Whoever the "them" is.

  14. Re: Professional Engineers have the power to say n by Grishnakh · · Score: 5, Insightful

    It's an American thing.

    Employers here don't care about holding onto skilled programmers or other skilled people, because PHBs think they can just hire replacements on a whim.

    Yes, in reality new ones are hard to find and take a while to get up to speed. The PHBs will even acknowledge this when they're trying to hire.

    But once they have one employed, they don't care about keeping him happy, because they think they're al interchangeable cogs.

    If you're seeing a giant disconnect here, yes, there is. This is how American corporations think; it makes no sense at all. I can't explain it. It's the same phenomenon where corporations will give a big salary offer to a new engineer, but once he's employed there, they'll just freeze his salary or give him paltry CoL raises, while giving new hires even bigger salaries, causing employees to switch jobs every 2-4 years (in Silicon Valley, it's 12-18 months).