Slashdot Mirror
How Did Volkswagen Cheat Emissions Tests, and Who Authorized It?
Lucas123 writes: The method by which Volkswagen diesel cars were able to thwart emissions tests and spew up to 40X the nitrogen oxide levels set by the Environmental Protection Agency was relatively simple. It was more likely no more than a single line of code used to detect when an emissions test was being performed and place the emissions system in an alternate mode — something as simple as a software "on/off" switch. Volkswagen AG CEO Martin Winterkorn, who stepping down as the result of his company's scandal, has said he had no knowledge of the emissions cheat, but software dev/test audit trails are almost certain to pinpoint who embedded the code and who authorized it. You can actually see who asked the developer to write that code," said Nikhil Kaul, a product manager at test/dev software maker SmartBear Software. "Then if you go upstream you can see who that person's boss was...and see if testing happened...and, if testing didn't happen. So you can go from the bottom up to nail everyone."
"However, not all companies follow detailed auditing processes. The primary reason, Kaul said, is the speed at which software is being released to the marketplace. It necessitates an "agile approach," resulting in millions of lines of code being worked on and checked into production every minute."
love is just extroverted narcissism
Supposedly the "cheat mode" is an extension of the "testing mode", where the car knows it is running on a Dyno because one set of wheels is turning at a high RPM and the other set are stationary. For a car with traction control this is normally a freakout event so they have to check for it and make sure not to go crazy just because the machine is strapped into a test harness. Once you have the otherwise required detection code in there, adding a single line to fully open the EGR valve when in that mode would be a piece of cake.
I read the internet for the articles.
Someone should have leaked this a looong time ago. Perhaps some dev, why not.
Hell it would have saved VW a lot of money! Think about recalling 1mln cars instead of 11mln!
Did VW really think it could get away with this indefinitely?
Fucking corporate morons...
I agree, but in real life things are often not this simple. You get requirements that you aren't sure are a good idea or the right thing to do, and you question them - but PHB assures you that it's all been approved and cleared by the people on higher floors, and you may even contact some of them and hear their agreement. You're a coder and now a lawyer, so...? You go ahead and write IF OBDIIPortHasSomethingConnectedToIt THEN EnterDiagnosticMode .
This case seems very egregious, but the truth of ethics in real life is often difficult to determine, and it's being thought about by a human whose livelihood may depend on the choice.
It's not hard to see how things like this happen. People will almost always act in (what they think is) their own best interest.
No flame here, just wondering. In my travels to Europe I haven't found them to be any worse pollution wise than American cities. Are these cars really that bad physically or are we talking goofy government crap?
Just asking.
Peace is easy to achieve, just surrender. Liberty is much harder get/keep.
I have been teaching UML, modeling and systems architecting at several companies that directly supply to the German car industry ( especially to Volkswagen and BMW ). It is the car makers themselves that impose rigid rules and constraints upon software traceability and configuration management. So the idea of
"software dev/test audit trails are almost certain to pinpoint who embedded the code and who authorized it"
is not that outlandish, and following such audit trails may well lead to (at least some of) the culprits.
Religous speak to God. Insane are spoken to by God. When all shut up, one can finally hear Shostakovich in peace
Sometimes but not always. This cheated the test both with and without a computer. It was instead detecting when it was on a rolling road. Emissions tests are always done with the car stationary but the wheels moving, and that'll be what the software was detecting.
From the sound it of it wasn't actually putting the car into a special 'mode'. It was turning on all the measures to reduce emissions. When the car was on a real road it was turning them off to get better performance and fuel consumption at the expense of emissions. So it sounds like the car does technically meet the regulations, but ignores them when it's on the road. So expect the recall to turn them on at all times, which'll mean you don't need the car/engine replaced, but will mean you pay more at the pump and see your car's less nippy than it was before.
Not sure why VW stock has tanked then...personally, this sounds like a very GOOD reason to go buy a VW car now...knowing when it comes to it, that it will really perform well!!
Light travels faster than sound. This is why some people appear bright until you hear them speak.........
Yeah, I worked for a large bank. I was told my task was to implement tax calculation code in javascript, so it would update on the page immediately. While I balked at the request, it was made pretty clear that either I do it or I would get fired. I was not given a javascript library that knew how to handle financial values. Javascript doesn't support integer-only values, so you're doing financial calculations with floating point roundoffs and errors. I happened to know about them but was not given any instructions that they were a problem. I wasn't given any instructions on how to make sure they didn't cause issues. As far as I know no more than basic testing was done on the code. I did get an email verifying that I had questioned it, but then I found out that all our emails are automatically deleted after 6 months or something like that. You get fired now, or you implement something dubious - what do you choose?
Why would anyone want to hide evidence? All the people seem to operate under the impression that this is some kind of big revelation, when in fact everyone in the industry knows that this is happening all over the place. This wasn't actually big news that Volkswagen is cheating, the big news is that some authority is finally taking the rules seriously, when for years both the european and US authorities did their best to look the other way when the evidence was shoved in their face. And if no one cares, why would you want to hide it?
Based on my experience with slimebag orgs and managers, it may have gone something like this:
A manager(s) asks for a software switch to deactivate the "clean" mode and also detect when smog testing is being done "in order to study and track resources devoted to environmental issues and make sure we understand and comply with the smog testing procedures."
Then a personal visit happens where key manager(s) ask the top-ranking technician to leave the bypass-on-test feature "on" in production. No paper trail. Experienced slimebags don't put such commands into writing.
When the IT lead later reveals "Executive X told me to in person", there's no written trail. It's one person's word against another's.
Sure, the IT lead is probably suspicious of the request, but when the big bosses tell you to do something, it's comply or hit the road.
I was once asked to cheat a client over database scalability. It was shortly after the dot-com crash, and knowing the market was really tight in Calif. and having a young family, it was a really difficult situation to be in. I won't go into the details here, but I was sick to my stomach over it. The experience made me more progressive.
Table-ized A.I.
From the article itself, (I know, I am not supposed to read the article), the traction control is mandated to be turned off when testing the emissions. So you already have a need to check that emissions testing is being done. Once that flag is set in the code other areas can use it and they may not even be aware of the ethical dilemma that arises. Perhaps someone was told to lower the fuel flow when the test_mode flag was on. Why would they feel that something does not "smell quite right" in that case? Someone else turned the flag on with a code change several years previously. There is plenty of other things being changed or turned off when in test_mode, so adjusting the fuel flow can fit right in with that. Unless they realized that test_mode was used to determine that emissions testing was being performed (it could be used for other tests also) and that the changes they are making will cheat the emissions test they might not even realize that something is wrong with what they are doing.
-- ssoorrrryy,, dduupplleexx sswwiittcchh oonn.. -Quote found on actual fortune cookie.
Yeah. I would not be too surprised if at some level in the organization this was sold as a debugging or trouble shooting measure, or some other benign reason was given for branching on detection of emissions tests.
Folks, you have to branch on emissions (and dyno) tests in the ECU solely because otherwise the safety side of things will bring everything to a halt. The most obvious reason (assume FWD) is that if the front wheels are going 65MPH and the rear wheels are going 0MPH, the traction control system is going to have a major freak out and say "HOLY CRAP WE ARE SKIDDING OUT OF CONTROL BRAKE FRONT UNTIL THE DIFFERENCE IS LESSENED". The procedure to enter dyno mode is not itself a secret.
So it's not a secret and it's not illegal for the ECU to detect and behave differently during the test, and everyone would know about this above-board feature. The secret-and-illegal part is modifying the behavior of things measured by the test while the test is running. That is a lot easier to keep secret and requires a lot less involvement from teams directly outside the module that is responsible for emissions.
[ Source: Tuner people that dyno fancy cars and have to solve these sort of issues. ]
The engineers who designed the engine - yes.
The engineers who coded the software for the engine computer? Why would they know what this does? The software is enabling a signal, hell the signal might even have a vague name, when a condition is met. The condition's name might not very clear.
Yes, a spec saying "when the car is undergoing a test then enabled the cheat mode to get past the test" would clearly implicate the developers.
But most likely it was: "when sig_x and sig_y and indicator_a are set, then set sig_Z to 1 in pattern P for n nanoseconds blah blah blah"
Someone knew what they were doing, and it probably went like this:
Engineer: We can't make this engine pass NOx tests.
Message goes up the chain to a certain decision making level, possibly the board. Marketing chimes in: We can't have this, we're already sending out teasers about our new urea-less engine technology, etc, etc.
Eventually a message comes down to fix it, in vague terms, entirely forgetting the original message that it's unfixable.
Engineers: struggle for ages.
In pub: Well, we could enable a special testing mode to pass the tests?
In work: Shall we do this -> up the chain. Original context is half forgotten. Approved.
Changes made. Software specs made. Timebomb implemented.
IANAL, but this is what I heard from an employment lawyer (paraphrasing): As long as you wrote and sent the e-mail, the onus is on the employer to show that they have replied to it in a reasonable fashion. Courts tend to throw out "I didn't see it" excuses and expect replies. And even if they replied verbally, oh well, too bad, they should've done so in writing.
Then you do the next best. You get it in writing from your immediate supervisor, playing as if you're verifying facts of the request. Someone that is used to receiving such requests from you. He/she, in turn, realizes what you are doing, and does the same with His/her immediate supervisor. This continues up the chain until it eventually reaches the upper management who requested it. If someone along the line doesn't do this, they become the scapegoat, not you.
It's a complete CYA action.
I find it odd to hear how programmers seem so abused by PHBs. Maybe it's an American thing, but in the UK, I've always found that employers want to keep hold of skilled people like programmers, because new ones are hard to find and take a while to get up to speed. This means that saying no is always possible.
(Nothing to to with official engineer status and ethics. There's no general requirement for engineer certifications for programmers here.)
I worked for a Small software house that made SAP type ERP software before SAP ate the majority of the market. This was 1998 or so...
We had a customer come to us and ask for certain modifications. Then a few more. Then a few more.
Not unusual, we made a lot of money from change orders. So the first few were done. All were acceptable in the Generally Accepted Accounting Practices guidelines.
Somewhere along the line the GAAP accountant realized that this last modification set would, taken in combination with all the other mods, make a check disappear from the system and become untraceable.
We refused to do it, and the customer dropped the product, saying we were too hard to deal with. A million+ of revenue were lost, no small amount for the company.
That customer? MCI Worldcom.
They clearly had picked apart the source code and found the edge case that triggered the behavior. I had left the company before MCI blew up, but my understanding is that they were called to give testimony/evidence in the trial.
This could be the same thing, a series of unrelated changes that trigger a diagnostic mode when hooked up to the test equipment.
If so, it would be very hard to trace who made the ultimate decision to do this, as it might be spread across many teams working independently.
Never answer an anonymous letter. - Yogi Berra