Slashdot Mirror
How Did Volkswagen Cheat Emissions Tests, and Who Authorized It?
Lucas123 writes: The method by which Volkswagen diesel cars were able to thwart emissions tests and spew up to 40X the nitrogen oxide levels set by the Environmental Protection Agency was relatively simple. It was more likely no more than a single line of code used to detect when an emissions test was being performed and place the emissions system in an alternate mode — something as simple as a software "on/off" switch. Volkswagen AG CEO Martin Winterkorn, who stepping down as the result of his company's scandal, has said he had no knowledge of the emissions cheat, but software dev/test audit trails are almost certain to pinpoint who embedded the code and who authorized it. You can actually see who asked the developer to write that code," said Nikhil Kaul, a product manager at test/dev software maker SmartBear Software. "Then if you go upstream you can see who that person's boss was...and see if testing happened...and, if testing didn't happen. So you can go from the bottom up to nail everyone."
Correction: "You can nail everyone that's in the official audit trail."
The people at the top that authorized it (or at least didn't condemn it) probably never actually sent a traceable e-mail to anyone. Nor did they touch any code. Nor do they appear in any meeting minutes. These sorts of discussions tend to happen over a drink in a bar somewhere, and for good reason.
It's cute how he thinks no one thought about this and sanitized the audit trail. I'm sure he also thinks his 4096-bit disk encryption thwarts even the most determined ne'er-do-wells.
#DeleteChrome
I *highly* doubt it was a single line of code. To toggle the car's "EPA Cheat" mode, maybe, but by all accounts, the system used a variety of inputs to detect artificial driving conditions (including, apparently, barometer data), as well as needing code to define what engine parameters to change once the mode was entered.
"However, not all companies follow detailed auditing processes. The primary reason, Kaul said, is the speed at which software is being released to the marketplace. It necessitates an "agile approach," resulting in millions of lines of code being worked on and checked into production every minute."
love is just extroverted narcissism
Then again, something similar might make a nice contest topic.
Someone should have leaked this a looong time ago. Perhaps some dev, why not.
Hell it would have saved VW a lot of money! Think about recalling 1mln cars instead of 11mln!
Did VW really think it could get away with this indefinitely?
Fucking corporate morons...
Most rolling roads don't spin the non powered wheels, so if the powered wheels are spinning and the others are stationary for any length of time its a good bet its having some sort of test. Obviously this isn't going to work with 4WD however.
No flame here, just wondering. In my travels to Europe I haven't found them to be any worse pollution wise than American cities. Are these cars really that bad physically or are we talking goofy government crap?
Just asking.
Peace is easy to achieve, just surrender. Liberty is much harder get/keep.
I have been teaching UML, modeling and systems architecting at several companies that directly supply to the German car industry ( especially to Volkswagen and BMW ). It is the car makers themselves that impose rigid rules and constraints upon software traceability and configuration management. So the idea of
"software dev/test audit trails are almost certain to pinpoint who embedded the code and who authorized it"
is not that outlandish, and following such audit trails may well lead to (at least some of) the culprits.
Religous speak to God. Insane are spoken to by God. When all shut up, one can finally hear Shostakovich in peace
If I were doing it I would have placed "// FIXME DEBUG" on that line of code. Like it was an internal testing mode which wasn't switched off, by accident of course.
EPA Cheat Code: Up, Up, Down, Down, Left, Right, Left, Right, B, A
"Never give up, for that is just the time and place when the tide will change." -Harriet Beecher Stowe ^_^
It's been a while since I watched my car being tested but do they hook up the car to a computer terminal of some sort? Could those be used to trigger test mode?
The test mode was triggered by monitoring which wheels were turning, position of the steering wheel, etc.
Basically they wanted to avoid the cost of installing a urea injection system so they cheated instead. Honda engineers were reported to be perplexed about how they managed to do this miraculous feat of engineering.
Here's a good article about what is known so far:
http://www.msn.com/en-us/autos...
What would be a big deal however is the code that detects whether the car is in test bench or on the road. Apparently it uses steering input and other such details. So that code block is the interesting part. Proper audit of the code changes, pull request authorizations would nail the engineer who actually did the dirty deed. But would there be code review meeting/minutes, comments fingering higher management?
This scandal will have some salutary effect in engineers who manage code, they would refuse to merge or pull such cheating code changes because it would leave their fingerprints indelibly for ever. They might even add comments in code covering their tails fingering the actual perp in the higher management.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
I wonder whether someone actually gave the order to implement a 'test defeat device' or they just started to optimize the engine to comply with regulation and to pass the test and then they went too far.
Someone in management made the decision to not install a urea injection system which is necessary to keep emissions to legal levels but costs a lot of money. Reportedly something like $400/vehicle. So it seems pretty clear that their "solution" to the problem was simply to cheat. This wasn't a case of optimization gone awry. They flat out knew what they were doing and went ahead with it anyway. As soon as they made the decision to not install urea injection, they effectively decided to cheat at that time because they were asking for the technologically impossible. There is no way they didn't know that their decision to leave off such a key piece of equipment would not result in unacceptable emissions. The engineers at VW aren't dumb. The decision was made for financial reasons (not surprising) but was aided and abetted by a bunch of engineers that should have known better.
The only real question seems to be who made the decision and who was responsible for executing it and covering it up.
I work for a company that has paid out some 30 Billion Dollars worth of fines to the US government. Where does that money go? I think it goes directly into the pockets of well-placed individuals, because we never hear about where that money goes.
When you pay a parking ticket, where do you think that money goes?
If telephones are outlawed, then only outlaws will have telephones.
I did get an email verifying that I had questioned it, but then I found out that all our emails are automatically deleted after 6 months or something like that.
Nothing prevents you from printing emails of instructions to implement dubious decisions. I've done this from time to time just to protect myself when I worked at a large company.
You get fired now, or you implement something dubious - what do you choose?
If it is clearly illegal or will be very likely to cause major problems then you should seriously consider walking. If it isn't so clear then you get them to document their instructions to you and you keep a copy (print if you have to) for your records to cover your ass should it be a problem down the line. Make sure you document your objections and make it clear that you have taken every reasonable effort to ensure that what you are doing is legal. If the decision is merely dumb but legal, same thing but don't worry so much about ensuring legality.
No “they” didn’t. One team wrote code to detect an emissions test. One team wrote code for a new emissions heavy efficiency high mode. One team wrote code to switch between emissions tests. One dude with a beard added if( TRUE === emissionsTest ) { mode = MODE_X43_Y ;} Thats all it took.
He is the boss, you work for him, for now, _not_ end of discussion.
He doesn't have to debate, but you don't HAVE to do it. Vote with your feet.
If you are making an Engineer's salary and can't afford to quit, you are a certified moron.
John McAfee 'It was like that time I hired that Bangkok prostitute; to do my taxes, while I fucked my accountant'
But frankly I have a VERY hard time believing that the engineers involved did not know that what they were doing would violate the law.
On the other hand, the code could be used to support a control test case, with the emissions controls switched on/off, to support development and how well the emissions controls were working on the same engine. This would allow the engine developers to test against a live performance benchmark. On the original hand, having that switch on/off automatically is quite hinky, so it's probably not (solely) for development testing.
It must have been something you assimilated. . . .
Professional Engineers have the power to say no and they have Ethics rules to fall back on.
Somebody knew, somebody high up knew, but I rather doubt that everyone on the engineering bench knew, and that means that they had to be fed plausible stories along the way.
Spare me. The engineers were the ones that eventually spilled the beans on this. They weren't fooled by some clever management strategy. They knew exactly what they were doing and they knew or should have known that it was illegal. While maybe not every engineer involved knew, more than a few certainly did without question. The engineers at VW aren't dumb. I know a few personally. Please stop with the attempts to find clever ways to not have to blame the engineers who were guilty of helping to commit fraud. Management may have ordered the crime but the engineers were the ones that carried it out.
The engineers who designed the engine - yes.
The engineers who coded the software for the engine computer? Why would they know what this does? The software is enabling a signal, hell the signal might even have a vague name, when a condition is met. The condition's name might not very clear.
Yes, a spec saying "when the car is undergoing a test then enabled the cheat mode to get past the test" would clearly implicate the developers.
But most likely it was: "when sig_x and sig_y and indicator_a are set, then set sig_Z to 1 in pattern P for n nanoseconds blah blah blah"
Someone knew what they were doing, and it probably went like this:
Engineer: We can't make this engine pass NOx tests.
Message goes up the chain to a certain decision making level, possibly the board. Marketing chimes in: We can't have this, we're already sending out teasers about our new urea-less engine technology, etc, etc.
Eventually a message comes down to fix it, in vague terms, entirely forgetting the original message that it's unfixable.
Engineers: struggle for ages.
In pub: Well, we could enable a special testing mode to pass the tests?
In work: Shall we do this -> up the chain. Original context is half forgotten. Approved.
Changes made. Software specs made. Timebomb implemented.
I find it odd to hear how programmers seem so abused by PHBs. Maybe it's an American thing, but in the UK, I've always found that employers want to keep hold of skilled people like programmers, because new ones are hard to find and take a while to get up to speed. This means that saying no is always possible.
(Nothing to to with official engineer status and ethics. There's no general requirement for engineer certifications for programmers here.)
I worked for a Small software house that made SAP type ERP software before SAP ate the majority of the market. This was 1998 or so...
We had a customer come to us and ask for certain modifications. Then a few more. Then a few more.
Not unusual, we made a lot of money from change orders. So the first few were done. All were acceptable in the Generally Accepted Accounting Practices guidelines.
Somewhere along the line the GAAP accountant realized that this last modification set would, taken in combination with all the other mods, make a check disappear from the system and become untraceable.
We refused to do it, and the customer dropped the product, saying we were too hard to deal with. A million+ of revenue were lost, no small amount for the company.
That customer? MCI Worldcom.
They clearly had picked apart the source code and found the edge case that triggered the behavior. I had left the company before MCI blew up, but my understanding is that they were called to give testimony/evidence in the trial.
This could be the same thing, a series of unrelated changes that trigger a diagnostic mode when hooked up to the test equipment.
If so, it would be very hard to trace who made the ultimate decision to do this, as it might be spread across many teams working independently.
Never answer an anonymous letter. - Yogi Berra
as there is a perfectly legal reason to detect an emissions test -- that the traction control and stability control doesn't go crazy.
In the UK it was reported on Newsnight (TV Programme) back in November last year. So it wasn't just engineers at VW that knew about it. What's surprising is the shit only hit the fan now.
It's fascinating to see how many posters here automatically assume that it must be the PHBs who pressured the engineers into this. Very few assume that the engineers saw an opportunity for a bonus or for the PHB to owe them one, and added the cheat function voluntarily. I've not seen any posts so far that suggest an engineer thought of the cheat and suggested it to a PHB.
A reminder that we tend to think of our peers as being much more ethical than "them" and look for reasons to think of them as victims of force or circumstances, and assume that "they" are only motivated by sheer callous greed. Whoever the "them" is.
And I find it hilarious that everyone here states that programmers and engineers work without requirements or documentation. I've worked places where the verbal meeting would have the engineer agreeing with everything, then when it's not written in the requirements document and signed off by 10+ people, it doesn't get built. Seems like all the programmers on Slashdot have never worked in a company larger than 10 people.
Learn to love Alaska
Professional Engineers have the power to say no and they have Ethics rules to fall back on.
But do they have the power to say no and keep their job, and keep their job without management making their work life miserable?
Engineer: We can't make this engine pass NOx tests.
Ah, but that wasn't the situation. The engine could pass fine, which is what was happening when the software detected the test conditions. The problem is when running like that the engine didn't have as much power or fuel economy as when operating in dirty mode. This hack was to make the car more appealing to consumers by (in a virtual sense) selling one car to the public, and having the EPA test a different one. This was pure deceit.
It's an American thing.
Employers here don't care about holding onto skilled programmers or other skilled people, because PHBs think they can just hire replacements on a whim.
Yes, in reality new ones are hard to find and take a while to get up to speed. The PHBs will even acknowledge this when they're trying to hire.
But once they have one employed, they don't care about keeping him happy, because they think they're al interchangeable cogs.
If you're seeing a giant disconnect here, yes, there is. This is how American corporations think; it makes no sense at all. I can't explain it. It's the same phenomenon where corporations will give a big salary offer to a new engineer, but once he's employed there, they'll just freeze his salary or give him paltry CoL raises, while giving new hires even bigger salaries, causing employees to switch jobs every 2-4 years (in Silicon Valley, it's 12-18 months).
While it is likely this was a sin of commission it remains plausible that no one did this at all. My thinking is that if instead of being programmed explicity the computer program was allowed to train itself for it's emission and performance tuning that a very natural outcome would be for it to learn to minimize emissions during emission type testing. Then on the test track it would learn performance and handling. etc... and so you end up with something that cheats but no one told it to nor was anyone even trying to make it cheat. It's just the result of getting what you optimize for.
One reason that I like that theory is that if you consider the opposite, that it was a conspiracy, then this is not the sort of thing you can keep secret easily. You might succeed but that's pretty hard especially considering the time span and the inevitable entry of new personnel and suppliers into the supply chain. So I don't think this was intentional. The exception might be if if it's a conspiracy of one. that for some reason there was just one guy who could pull off everything. THen you would have a shot of keeping this secret.
Some drink at the fountain of knowledge. Others just gargle.
This is Germany, right? Back in the olden days the culture was if someone on high told you to do something (turn on the gas) you did it no questions asked. Maybe that culture still exists (following directions without asking questions).