Slashdot Mirror
Volkswagen Diesel Scandal Logistics Imply Sizable Conspiracy
Guinnessy writes with an interesting analysis of the Volkswagen software cheating scandal: Physics Today's Charles Day takes a look at how diesel engines work, and why it's clear it's not just a lone software engineer who came up with the cheat. "...[S]oftware is impotent without hardware. To recognize when a car was being tested and not driven, the defeat device required data from a range of sensors -- sensors that a noncheating car might not need.... Whereas it's conceivable that a single software engineer, directed by a single manager, could have secretly written and uploaded the code that ran the defeat device, installing its associated hardware would require a larger and more diverse team of conspirators," he says.
I've worked as a partner for some car companies in both the US and the EU, and I know for a fact that the firmware that goes into their control systems is very tightly controlled, requiring sign-offs from senior execs for design and feature changes. There's no way code this critical could have simply been dropped in by some R&D leads. No. Way.
If you put any thought into this at all, you realize it is a massive conspiracy. Other automakers add expensive, space consuming devices to eliminate NO pollution. These is no way a single programmer could have made a change and all the engineers would go "Look, we don't need all the extra hardware, it passes the test!" Lots of people would notice immediately during the design phase.
The linked article makes the point that the sensors and hardware would not be necessary. I think the writer seriously underestimates to what extent a modern car with protection systems will try to juggle different constraints. Things like non-driving wheel rotation (defeated by being on a lab stand) are needed for breaking systems and possibly to some extent to moderate throttle control for stability. Wheel movement patterns are also needed and useful, even if you don't actually have electric power steering.
Regulating the exhaust gas recirculation somehow also makes sense. You might go totally on and off, but you would certainly want to keep it at a sensible level. You want good acceleration and full combustion of fuel while still not emitting to much nitrous oxides. It makes total sense to me that you might want to design your control system to try to judge not only the current emission levels, but also the overall driving pattern (steady straight ahead, repeated stop and go, etc) with some kind of state machine to try to find the best EGR regulation regime. This requires sensors and ways to regulate the feature.
My most innocent guess about how something such as this might have happened was an intent to find a good regime that would give nice bursty performance, while keeping nitrous oxides low overall. Progressively, the control regime was pushed until it ended up in the corner where the case of EGR being properly activated under real-world conditions basically does not happen. Some parts of it might even in the end be a bug between the intended state transitions and the actual ones. Like all bugs that give performance that seem too good to be true on the metrics you really care about (fuel consumption and enjoyable driving), no-one investigated.
Do I think it happened this way? It's hard to say. Probably not. But, in one way, it's even more frightening than an evil conspiracy. It's easy to say "I wouldn't take part in a conspiracy by my employer". It's harder to say "I would never be pressed to write code with goals that could not be fulfilled, eventually find a hack that seemed to work, and maybe ignore investigating why it worked so well"...
At what point does a group of people, perhaps thinking they're working to create something good, but that actually results in something that maybe isn't so good, become a "conspiracy"?
Let's look to open source efforts like GNOME 3 and Firefox 4 (and later versions). Here we have well-established software products, with many users, and extensive communities built around them. While they'll take outside contributions, stewardship of such projects is quite tightly controlled. Yet at some point, very bad decisions start being made by the developers of the products, and not necessarily in bad faith. There comes a point where some influential members of the community think it's important to target "average users" or "mobile users" or perhaps to compete with a similar product from another vendor by imitating it.
But by doing so, they end up completely trashing their own products. A desktop environment like GNOME 3 becomes almost completely unusable on the desktop by power users, who make up the bulk of its community. A browser like Firefox throws away an intuitive and usable UI for one that's nonsensical in most ways, while long-standing performance problems, resource usage excesses and bugs remain unfixed.
Yes, many people can be and are involved in such debacles. But do we know that they were all acting maliciously? Do we even know that they actually knew what they were doing? Somebody pushing for Firefox's awful Australis UI, for example, may have thought he was helping design a good, novel UI. But rational outsiders and Firefox users thought very differently, clearly. That's why Firefox is now at only about 7% to 8% of the browser market, when it used to be above 30%.
We can't deny that GNOME 3, and Firefox version 4 and later, have been project-level failures involving many people. But despite the negative and unwanted outcomes, it's difficult to say with certainty that there was any sort of "conspiracy" involved. It could very well be people working together in good faith, who unfortunately only end up creating a very awful outcome.
Modern cars use a system to stabilise the car in the event that one or more wheels starts to lose adhesion - commonly called things like ESP/DSP/ESC
The car wants to know when it's on a dyno or other testing device where only one set of wheels move, and the others do not - if this were NOT the case, it would assume that the rear wheels have lost adhesion with the road, and will serious interfere with the power provided to the front wheels.
So "the defeat device required data from a range of sensors -- sensors that a noncheating car might not need" is totally and utterly rubbish, it likely needs a single line of code like this:
> if(EngineMode.Test){ ... do something to improve emissions ... }
Furthermore, many cars may already have a "very low emissions" mode or similar - there may not be a "special" mode specifically for EPA tests which a different profile for timing, fuel injection etc. - the cars computer essentially changes the "configuration" of the engine on the fly, based on driving conditions, driver input, gear, fuel quality, engine feedback etc - and it does all this during NORMAL operation.
If a "high efficiency / low emissions mode" already existed, then the code could be further reduced to
> if (EngineMode.Test ){ Engine.PerformanceProfile = LowEmissionsProfile }
Of course, it's unlikely that there would be a high level language available to engineers to make it quite so readable as above - but hopefully the code illustrates the point.
FWIW I strongly suspect that the "low emission profile" in place here in VW *IS* a "special" doctored one to fool emissions tests, but the detection of actually being in a test? Probably already existed.
I think the explanation as to why diesel engines create more nitrogen oxides and how the EGR works was simple and on point, but the conclusion not so much. I drive a diesel myself, but it is a 2006 model, it doesn't have adblue injection, my exhaust system only has a catalytic converter and a particle filter (and an EGR, of course). Even though it is an old model, like most cars since then it has more than enough sensors to do what VW did: individual wheel speeds for the ABS, steering wheel angle for the ESC, multiple sunshine sensors, front and rear suspension angles for the headlight height control, multiple temperature and pressures sensors on both the intake and exhaust, multiple flow rate sensors, mass air flow sensors, multiple sensors in the cooling system etc.
That's why I find the article a bit thin on new information, I'm certain the embedded engineers at Bosch/Delphi/Siemens/etc. could have done that with far less information that a more modern car has.
Did they all knew about it? Probably. Did they made hardware efforts to cheat? I don't believe it yet, that's the point of cheating, "passing" the test without having to add new hardware, there is plenty of data that can tell you if the car is really moving or in a test chamber.
Anyone who actually works in the auto industry is pretty much certain this wasn't a lone-wolf operation. I know because I've been in the industry myself for a good chunk of my career including right now. This is very much the water cooler talk right now and nobody believes it was just one or two guys. I run a company that makes wiring harnesses and many of our products go into automobiles made by the Big 3. There are WAY too many people and groups involved in the engineering, design and testing and manufacture of these cars for this to be pulled off entirely in secret. While it would not have been known across the company it would have had to have been signed off on by more than a few including engineering, management and probably testing as well.
This was not done by accident. It was not done by some poor engineer asked to do the cheat on pain of losing his job. This was an intentional and premeditated fraud and it isn't the first time something like this has happened. About 15 years ago a bunch of truck manufacturers including Volvo and Caterpillar were caught doing something similar. Probably won't be the last time we see it either given the amount of money at stake. While I'm sure VW is probably going to try to throw some low level people under the figurative bus, I'd be shocked if this didn't go pretty far up the food chain. Maybe not all the way to the top but probably up to the heads of engineering and R&D at the least. I can't imagine how the engine designers and their management team wouldn't know. This stuff isn't magic and questions would be asked for which there is no satisfactory answer via software.
Many sports cars have at least two performance settings...Mustangs with their Black key and Red keys for instance. How stupid would that be if it didn't change the performance of the engine? And when you do that for more power, you are undoubtedly going to get worse mileage and emissions.
So what does the EPA do? Test the cars in "normal" mode and assume that no one will really ever use the "sport" mode? Reality says almost everyone will be in sport mode all of the time.
When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
To recognize when a car was being tested and not driven, the defeat device required data from a range of sensors—sensors that a noncheating car might not need.
At what point does a group of people, perhaps thinking they're working to create something good, but that actually results in something that maybe isn't so good, become a "conspiracy"?
The moment it becomes obvious that what they are attempting is impossible and they start looking for illegal ways to circumvent a test. At that precise point they should have stopped and done something else.
There is no real grey area here where people weren't fully aware of what they were doing and at no time were they under any illusion about the legality. The people who implemented this are professional engineers who knew(or should have known) what the rules were and decided to go ahead anyway. This isn't a piece of consumer software where there are no federal laws involved. This wasn't a piece of software where what seemed like a good idea ultimately didn't work. No, they intentionally and with premeditation committed this fraud. Stop it with trying to excuse what they did.
I think that you missed the point of the previous post. It could be that many people involved thought that they were adding a performance function.
I did not miss the point. The point was wrong. They did not think they were adding a performance function. That's not how it this stuff gets developed. They would have known if this idea worked or not before it left R&D.
Granted, at some point it clearly crossed the line.
And that is where they should have stopped. No equivocation is necessary. The moment they realized it was illegal/impossible they should have stopped. It was reasonable to try to come up with a clever way to avoid the cost of adding a urea injection system but the would have known if this was feasible before the idea left the R&D lab. Once it got to the production engineers, there is no possible way they didn't know that what they were doing.
The auto companies repeatedly test the cars on their test bench. They use specially instrumented engines that collect so much of data on those test cars. Knowing which data was collected on the test harness and which were real road data is a legitimate data for debugging and fine tuning. The amount of data collected (actual valve position, commanded valve position, sensed crank angle, actual crank angle, time fuel injector open, time fuel injector done, blah blah ...) would so copious they might turn off certain data collections under certain circum stances.
They might have started with a special manual switch on the dash to turn on "test bench" mode. They forget to turn it on a few times, invalidating lots of collected data because "on test bench" field was wrong. Some clever guy suggest automating it. All these would be very legitimate and most engineers on the team would be working on good faith that it is not a cheat device.
What I am trying to say is "auto detection of test bench run" has a legitimate purpose. They also have so many use profiles. CA air standards profile, Euro air standards profile, China air standards profile, India air standards profile etc. All these use cases are also quite legitimate. All the engineers working on all these projects would be doing work without compromising their integrity or ethics.
Eventually someone high up had a clever idea to load China/India urea use profile when the car is not on test bench. This work does not involve company wide collusion. It would require very few engineers on the coding side and a few top level managers. They would know what they are doing is implementing a cheat device. They might have even done it as a stop-gap measure intending to correct in a few weeks or few months.
Some scenario along the lines of ... "Heinz, the air-quality team needs a few more weeks, they are behind, we are going to miss the deadline. But they are close, just a few more weeks. For now let us load India profile when not on test bench, once the air-quality team finishes the project we can quietly restore the setting. Or we have to delay the ship date by a few weeks". "Erwin, are you sure they would be done in six weeks". "Definite, absolutely". "OK I will talk to Walter and Karl. Keep Adolf and Joseph out of the loop. Keep it under your hat, and make sure there is no paper trail".
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
"the hand breaking system"
Wow, I'd try to stay away from that! Why would they put such a thing on a car?
Only sensors required to determine which wheels are moving are wheel speed. Any ABS equipped car needs those for at least 3, and usually 4 wheels so that ABS works properly. Does not require conspiracy to put in extra hardware.
If he'd bothered to speculate, he probably would have realized he was wrong. Day appears to be good at summarizing physics, but cars are more complicated than they appear at first glance.
Anti-lock brakes, computer controlled transmission shifting, variable assist power steering, fly-by-wire throttle and closed loop engine management all require sensors. Taken together, those sensors exceed what's needed to explain VW's cars ability to distinguish between active driving and a steady state test.
Don't worry, this is Germany so something WILL happen to people at the company. This investigation won't be swept under the carpet.
I have lived in Germany for many years and don't believe that Germany is better than other western european countries in this regard.
They may have known, but what was the alternative? Get fired, and in a manner that ensures they will never work in their field again?
The alternative is that you don't commit a crime. Why is that so hard to understand? This was FRAUD, plain and simple. If my boss comes to me and asks me to commit a crime so the company will make more money my answer is to gather my personal effects and seek employment elsewhere.
We are not talking about engineers who lacked options. The auto industry isn't one where they can get blackballed from every working again. These are well paid, educated people who knew (or should have known) what they were doing and decided to commit a crime.
Or go to the regulator and media, bring down the responsible parties, and get sued so hard their grandchildren will be paying the lawyer bills?
You can do that OR you can just leave. Either option is better than committing a crime.
It would only really require one or two people to pull this off. All the necessary components are innocent enough. (Also, the idiot who wrote the article is full of bull about "requiring additional hardware".) Components (with innocent purpose):
* Hardware necessary to detect testing mode: all cars have a speedometer
* Software to detect testing mode: reasonable to use for internal tests, and on production for traction control
* Hardware to allow software control of EGR: necessary for efficiency
* Software to adjust NOx/performance/efficiency levels: legitimate to have various modes, or for use in areas with different pollution laws
It would be trivial for one guy to write the code to have low NOx during testing, and high efficiency/performance otherwise. However, half the company would have to know they were cheating.
Don't waste your vote! Vote for whoever you want, unless you live in a swing state it won't matter anyways