Slashdot Mirror


Volkswagen Boss Blames Software Engineers For Scandal (bbc.co.uk)

hattig writes: Today VW's Michael Horn is testifying to Congress and has blamed the recent scandal on engineers saying: "It's the decision of a couple of software engineers, not the board members." However, 530,000 cars in the U.S. will need to be recalled for significant engine modifications, not a software fix. Only 80,000 Passats are eligible for the software fix. There is no word on the effects these modifications will have on the cars' performance, fuel consumption, etc. The BBC reports: "The issue of defeat devices at VW has been a historic problem, points out a Congress panel member questioning VW US chief Michael Horn. In 1974, VW had a run-in with US authorities regarding the use of defeat devices in 1974, and in December 2014 it recalled cars to address nox emissions."

54 of 479 comments (clear)

  1. Cultural? by fyngyrz · · Score: 5, Insightful

    The previous events seem to point towards a problem in the company's culture, rather than just a couple engineers. Maybe I'm too cynical. But that's what it "smells" like.

    --
    I've fallen off your lawn, and I can't get up.
    1. Re:Cultural? by tnk1 · · Score: 4, Insightful

      Perhaps. Although there is always going to be a temptation to cheat on things like tests.

      Still, process and checks should have been able to catch this. It may be that the engineers did it, but the managers failed to enact a process to catch it because it was overhead and not important.

    2. Re:Cultural? by jellomizer · · Score: 5, Insightful

      I need to agree. Germans take a lot of pride in Engineering as a culture. To say the German Engineers took short cuts just to pass US tests seems more unlikely than a strict Wink-Wink-Nudge-Nundge from the Bosses to the engineers with the side effect of or-else.

      --
      If something is so important that you feel the need to post it on the internet... It probably isn't that important.
    3. Re:Cultural? by prefec2 · · Score: 5, Interesting

      Actual the software was from Bosch. However, the Bosch guys said it would be illegal to use this specific feature. Looks like someone did not get the memo ;-)

    4. Re:Cultural? by mhkohne · · Score: 5, Insightful

      I believe the old saying 'A fish rots from the head' is applicable here.

      --
      A thousand pounds of wood moving at 300 feet per minute. Don't get in the way.
    5. Re:Cultural? by Austerity+Empowers · · Score: 5, Insightful

      Which is how I have seen it done elsewhere. If your company follows the Jack Welch style "fire the bottom 10%" mandate, and you are the guy that refuses to stand up for things like principles, guess who is going to be related "below expectations"? It's not just speculation, this sort of thing really happens.

      Then someone gets in trouble, they blame someone irrelevant, "fix" the problem (that was discovered) and drive on. Meanwhile massive cheating, lying or intentional ignorance continues to happen on other things.

    6. Re:Cultural? by soft_guy · · Score: 5, Insightful

      More likely managers ordered the engineers to do it because of pressure from higher up.

      --
      Avoid Missing Ball for High Score
    7. Re:Cultural? by Daemonik · · Score: 5, Insightful

      The engineers did it because the CEO on down told them to make it happen or go find another job. There were far too many sensors on the car involved with this cheat for anyone to believe it was just a few rogue engineers.

    8. Re:Cultural? by PPH · · Score: 4, Informative

      Bosch guys said it would be illegal to use this specific feature.

      Forbidden!

      --
      Have gnu, will travel.
    9. Re:Cultural? by nmb3000 · · Score: 5, Insightful

      I need to agree. Germans take a lot of pride in Engineering as a culture. To say the German Engineers took short cuts just to pass US tests seems more unlikely than a strict Wink-Wink-Nudge-Nundge from the Bosses to the engineers with the side effect of or-else.

      Completely agree. Not to mention that most engineers work to a functional specification. The software controlling what the emissions control computer reports is a pretty simple concept: pull readings from the on-board sensors and push them onto the output bus. Anything that deviates from that would need to have been driven explicitly by somebody. Code that detects emission testing equipment and conditions doesn't just get added by a couple of engineers on a whim.

      I'm sure that a program manager was given the EPA requirements and told "You must meet these (by any means)." That PM passed them on to the engineering team with clear instructions that the limits must be met, one way or another.

      --
      "What do you despise? By this are you truly known." --Princess Irulan, Manual of Muad'Dib
      /)
    10. Re:Cultural? by Anonymous Coward · · Score: 5, Insightful

      That's why You always ask for such orders in writing. And always make copies. Bureaucracy is the process of constant preparation for an eventual litigation.
      If You don't get the orders, get out while You still can, because You WILL be held responsible for it. Be happy if You only get sacked, and not sued into oblivion.

    11. Re:Cultural? by bobbied · · Score: 5, Insightful

      Having been the guy who got made into the scapegoat and got a swift kick out the door, I can tell you this is true in some places. I didn't get fired, but they sure made it in my best interest to leave, ostensibly because I was set up to fail by the process and I failed to realize they really didn't want to fix the problem soon enough. Being true in some places does not make it universal. MOST places I've worked actually made it a point to accurately find and fix "problems" as they came up and didn't waste the time and effort necessary to find the scapegoat to blame in a sea of CYA documents.

      I suspect that VW just doesn't have the corporate culture of ethics over profit, at least at some level. What's happening now is they are in the midst of figuring out exactly what happened. Who did what, who authorized what and who can CYA the most effectively. Problem here is that *somebody* or a group of *somebodies* broke the law in a really big way and there is a real risk of being walked out of the building in handcuffs. This is when corporate lawyers start echoing the standard refrains of "Don't destroy any records", "where is your search warrant" and "don't talk to investigators or the press without a lawyer present" lines to everybody.

      Somebody is likely going to jail, or at least facing criminal charges in both the EU and the USA.... Expect there to be a lot of finger pointing from here on out.

      --
      "File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
    12. Re:Cultural? by Etherwalk · · Score: 5, Insightful

      That's why You always ask for such orders in writing. And always make copies. Bureaucracy is the process of constant preparation for an eventual litigation.
      If You don't get the orders, get out while You still can, because You WILL be held responsible for it. Be happy if You only get sacked, and not sued into oblivion.

      Depending on the corporate structure, you doom your career with the company if you ask for such orders in writing.

    13. Re:Cultural? by tgatliff · · Score: 5, Interesting

      Bosch Egrs wrote and tested the software and not VW. Someone at VW requested the change and paid Bosch to do it. Also, Bosch later sent a letter to VW telling them it would be illegal if they deployed this software. So apparently just a couple software engineers at VW not only write software but also come up with "innovative" ideas on companies largest project, manage their largest vendor, perform project management meetings with that vendor, and receive/answer the physical mail. No wonder they they made "mistakes". Other than attending board meetings, those couple of engineers basically run the entire VW company!!

    14. Re:Cultural? by viperidaenz · · Score: 3, Interesting

      None of them broke any laws, so what are they guilty of?
      They're software engineers in Germany.

      The laws being broken are USA laws for importing and certifying cars. The engineers didn't import anything in to USA.

    15. Re:Cultural? by Chris+Mattern · · Score: 3, Interesting

      Reminds me of the grape juice concentrate that used to be sold during Prohibition with step-by-step instructions on how to ferment it into wine--solely so you could avoid accidentally breaking the law by doing it, of course.

    16. Re:Cultural? by Jack+Griffin · · Score: 4, Insightful

      Depending on the corporate structure, you doom your career with the company if you ask for such orders in writing.

      If you are pursuing a career in a company that you know behaves in such ways then you get what you deserve. Give it to me in writing or I'm happy to find a job elsewhere.

    17. Re:Cultural? by nblender · · Score: 4, Informative

      Ontario has already announced that you will be unable to register your vehicle next year until you provide documentation showing you've had the work done.

      http://www.kitchenerpost.ca/news-story/5951438-ontario-vw-owners-could-be-forced-off-road-without-clean-up/

    18. Re:Cultural? by lgw · · Score: 4, Insightful

      The hardware fix may well be a urea injector, like previous models used. That wouldn't have any material effect on performance or fuel economy. If the cost of adding it is paid by VW, there's no reason not to.

      When the fix is mostly hardware modification, it's hard to blame the problem solely on software engineers!

      --
      Socialism: a lie told by totalitarians and believed by fools.
    19. Re:Cultural? by magarity · · Score: 5, Insightful

      You don't have to outright ask for orders in writing; just notify manager via email of the progress you're making on the emissions cheat project and ask an innocent question. Print out and keep the reply as proof management is complicit.

    20. Re:Cultural? by Registered+Coward+v2 · · Score: 3, Interesting

      This is when corporate lawyers start echoing the standard refrains of "Don't destroy any records", "where is your search warrant" and "don't talk to investigators or the press without a lawyer present" lines to everybody.

      Somebody is likely going to jail, or at least facing criminal charges in both the EU and the USA.... Expect there to be a lot of finger pointing from here on out.

      I agree withe everything up to "don't talk to investigators or the press without a lawyer present." Never forget that a corporate lawyer doesn't represent you, he or she is their to protect the corporation and will throw you under the bus at the first opportunity. Anytime a lawyer is sent to "help" the first thing to ask is "Who you represent?" followed by "are you my lawyer?" and "is everything we say confidential and privileged?" if the answer to the last two isn't yes and yes they are not on your side.

      --
      I'm a consultant - I convert gibberish into cash-flow.
    21. Re:Cultural? by Registered+Coward+v2 · · Score: 4, Funny

      I'm waiting to see what the German perp walk looks like.

      https://www.youtube.com/watch?...

      --
      I'm a consultant - I convert gibberish into cash-flow.
    22. Re:Cultural? by painandgreed · · Score: 4, Insightful

      Depending on the corporate structure, you doom your career with the company if you ask for such orders in writing.

      Never ask, just do it and send them back an email saying "I did X as you instructed me to but the problems Y and Z are still there, do you want me to do anything about it?" to create a paper trail.

    23. Re:Cultural? by pkinetics · · Score: 4, Insightful

      That assumes you wouldn't get blackballed in the industry as a troublemaker, or some other subtle keyword passed back and forth thing.

    24. Re:Cultural? by Copid · · Score: 3, Interesting

      Is this the one place where embedded automotive code has no traceability? Nobody knows who checked in the mods, who approved the changes, and whether that approval is traceable back to a defect/requirement/change request? Was I totally misunderstanding how the automotive industry handled its microcontroller firmware? It seems to me "Where did this code come from, who approved it, and what was the justification?" should be just about the easiest questions in the world for this type of engineering shop to answer. Maybe things could get sketchy when the manager who approved it points a finger at unwritten orders from his superior, but until that point, the paper trail should be completely clear. Right?

      Right?

      --
      An interesting anagram of "BANACH TARSKI" is "BANACH TARSKI BANACH TARSKI"
    25. Re:Cultural? by mjwx · · Score: 4, Insightful

      More likely managers ordered the engineers to do it because of pressure from higher up.

      Bosch has openly said that they warned VW about the code in their ECU's being illegal in 2007. The VW management don't get to plead ignorant on this.

      --
      Calling someone a "hater" only means you can not rationally rebut their argument.
    26. Re:Cultural? by Alain+Williams · · Score: 3, Informative

      You haven't met many managers have you ?

    27. Re:Cultural? by Pubstar · · Score: 4, Insightful

      I work as a contractor for the goverment. Anytime a Civilian gives me an order I know is wrong over the phone or in person, I email them paraphrasing what was said and ask a stupid question about it. I then use that email to forward to my Civ POC to get things sorted out. Its been a live saver several times.

    28. Re:Cultural? by The+Grassy+Knoll · · Score: 5, Informative

      It did in the building industry

      [snip]For 16 years the Consulting Association compiled a secret database on thousands of construction workers[/snip]

      --
      They will never know the simple pleasure of a monkey knife fight
  2. Uh huh. by KermodeBear · · Score: 5, Insightful

    Yeah, I'm sure, a few rogue software guys got together and said, "Gosh, how can we cover for the people who built the engine that isn't as efficient as it is supposed to be? Surely there's no legal ramifications for cheating on federal emissions tests!"

    It doesn't make sense on too many levels. What a bunch of crap.

    --
    Love sees no species.
    1. Re:Uh huh. by msauve · · Score: 4, Interesting

      "Yeah, I'm sure, a few rogue software guys got together and said"...

      More like - "The beancounters won't let us add a few hundred Euro in hardware so we can pass emissions tests, and the boss promised us a large bonus if we can do it with software. Hey! I've got an idea."

      --
      "National Security is the chief cause of national insecurity." - Celine's First Law
    2. Re:Uh huh. by bhv · · Score: 5, Interesting

      Think a little bigger:

      "The two men, Ulrich Hackenberg, Audi’s chief engineer, and Wolfgang Hatz, developer of Porsche’s Formula One and Le Mans racing engines, were among the engineers suspended in the investigation of the emissions cheating scandal"

      I doubt these gents have been software engineers for a long time.

      Source: http://www.wsj.com/articles/vw...

    3. Re:Uh huh. by Anonymous Coward · · Score: 5, Interesting

      Are you suggesting only hiring psychopaths as managers? If you eliminate both the sociopaths and the psychopaths, where are you going to find managers?

    4. Re:Uh huh. by nytes · · Score: 5, Insightful

      We had a president at our company who had monthly meetings with all the managers where they had doughnuts and just chatted.

      One meeting he asked everyone to be candid and say if they had any concerns about upper management or the direction of the company. The following week, everyone who spoke up at that meeting was fired.

      --
      -- I have monkeys in my pants.
    5. Re:Uh huh. by Grishnakh · · Score: 4, Interesting

      Actually, I think part of the problem is that either they didn't have enough space available for the extra emissions hardware needed to do it right (urea injection), or it would have driven the cost up too much. The cars this affects are their smallest cars, which don't have much extra space in the engine bay, and probably have a lower profit margin as well; on larger models like SUVs, this stuff probably isn't a problem.

    6. Re:Uh huh. by Grishnakh · · Score: 4, Informative

      It's not about efficiency, it's about emissions, which runs counter to efficiency. It's the whole problem with diesel engines to begin with: if you want better fuel economy, you have to increase combustion temperature. Doing so with diesel gives you higher NOx emissions. So you have to lower combustion temps to keep those down, but then you sacrifice mpg and also horsepower (both of which are very important to drivers for fairly obvious reasons).

      If the company could, they'd maximize fuel economy and power and ignore emissions, but that would give you huge NOx emissions, which causes serious smog problems (Paris has much worse smog than most American cities from what I hear, because of all the diesel engines).

      There really isn't a great solution to this it seems. Urea injection is supposed to help a lot though. But it seems like the best answer is to give up on diesel for small passenger vehicles and stick with gasoline or just move to EVs.

  3. Interesting by The-Ixian · · Score: 5, Insightful

    That company executives rarely know what is going on in their organization.

    What do they get paid to do again?

    As an executive, you take on the responsibility and risk for your department/BU/company/team/whatever and the people under you. *That* is why you get the big bucks, not for any other reason.

    If somebody you are responsible for screws up, it is YOUR JOB to know about it!

    --
    My eyes reflect the stars and a smile lights up my face.
    1. Re:Interesting by zlives · · Score: 3, Insightful

      thats why the poor CEO quit, even though it clearly was not his fault...

  4. Oh, bullshit by Trailer+Trash · · Score: 5, Insightful

    Software engineers have little natural incentive to make the car perform differently for testing than for regular use. If the car is incapable of meeting emission standards without this sort of hack then that's an issue for the mechanical engineers, not the software guys. There's no reason to believe this was the result of anything but orders from on high.

    1. Re:Oh, bullshit by trout007 · · Score: 3, Insightful

      As a mechanical engineer this is pretty simple. The efficiency of an engine is related to the temperature difference. To get higher efficiency in a diesel you need higher compression. The problem is at high temperature you get NOx formation. There is nothing wrong with the mechanical design. They are near what is possible with thermodynamics. Im sure the ME's reported to the bosses if you want this efficiency and power you won't meet the emissions. Someone higher up made the call to cheat.

      --
      I love Jesus, except for his foreign policy.
    2. Re:Oh, bullshit by Geoffrey.landis · · Score: 5, Informative

      http://www.wsj.com/articles/vw...

      That article explains a lot.

      So, VW hired a top engineer away from Daimler to revamp the VW line. He brought in clean-diesel technology ("BlueTec") licensed from Daimler, but the engineers at VW hated the idea of licensing technology from a rival, because they said they could do just as good with the turbocharged direct injection designs that they'd been working on for years. Nevertheless, VW went through with an engine design with the licensed BlueTec, made a prototype engine... and then the CEO got pushed out, the chief engineer got pushed out a month later, and the new CEO put the engineers who'd opposed licensing outside technology in charge of making a new VW clean-diesel engine and cancelled the license from Daimler. So, they had essentially doubled down on the bet that they could do just as good on efficiency and NOx emissions without licensing the Daimler BlueTec, And right as they did that, the new CEO announced ambitious targets for selling clean diesels in the US.

      The story is beginning to make a bit more sense now.

      --
      http://www.geoffreylandis.com
  5. Bullshit ... by gstoddart · · Score: 5, Insightful

    Aren't there actual mechanical parts of the engine which simply weren't even implemented and then this kludge was done in software?

    You can't design this way of cheating without people who know the details of the engine signing off on it.

    This is so much bullshit it isn't funny.

    A software engineer could not have made the decision to leave off the components which were supposed to make clean diesel.

    This is purely about finding a scapegoat.

    --
    Lost at C:>. Found at C.
    1. Re:Bullshit ... by Doke · · Score: 5, Informative

      Yes. They omitted the diesel exhaust fluid (urea) injection system. I heard it saved about $400 per car.

    2. Re:Bullshit ... by gstoddart · · Score: 5, Insightful

      Which means there is no defensible way that you can say this was purely a software kludge designed to hide some information.

      I'm pretty sure there were a lot of people who simply HAD to be actively involved in this decision.

      This is a straight up lie, and the people making it know it is.

      --
      Lost at C:>. Found at C.
    3. Re:Bullshit ... by gstoddart · · Score: 5, Insightful

      No, the point (made many times already, try googling for once)

      Oh go fuck yourself.

      VW lied about how they achieved these numbers, and are claiming a couple of software engineers are the culprits.

      So, yes, actual mechanical parts they never implemented and then lied about, and now they're looking for a scapegoat.

      The people responsible for the engine design pretty much had to know this. Blaming it on software engineers is an outright lie.

      They lied about how they did this, they lied about how they faked it, and they're lying about who is at fault. The only "clever design" was systematic fraud.

      --
      Lost at C:>. Found at C.
  6. yes, that sounds reasonable by Pseudonymous+Powers · · Score: 5, Funny

    Fritz: Hey, Hans, you know how we are both software engineers working for Volkswagen?

    Hans: What a strange question, Fritz. But yes, I suppose I do know that.

    Fritz: Well, I was thinking, these new U.S. emissions standards are actually pretty stringent, and I don't think our diesels can pass them.

    Hans: Yes, this is obvious. So?

    Fritz: Well, what if we changed the software so that, while the cars were being tested, they behaved in a completely uncharacteristic way so that they could appear to comply with the standard?

    Hans: You mean if we wrote a test-detection and -subversion routine into the car's firmware?

    Fritz: Yes, of course.

    Hans: But how would we personally stand to benefit from that?

    Fritz: Well, we'd be able to sell more cars in America that way.

    Hans: We? You mean Volkwagen. Sure, until they caught on. But Fritz, we're just engineers--we get paid the same either way.

    Fritz: Well, we could tell the executives about it later, and maybe they would reward us.

    Hans: No, trust me, the executives won't want to know about it.

    Fritz: Yes, they do certainly value integrity over the bottom line. Completely unlike an engineer. Oh well. I guess we'll just have to do it without telling them, and for no good reason at all.

    Hans: Yes, that sounds reasonable.

    1. Re:yes, that sounds reasonable by Qzukk · · Score: 5, Funny

      Fritz: We should at least tell Steve over in the engine construction department that he won't need to install urea tanks on all of these cars, right?

      Hans: Yes, this is good. He will surely share his bonus with us and not throw us under the bus when people discover what we have done.

      --
      If I have been able to see further than others, it is because I bought a pair of binoculars.
  7. Michael Horn is a poor leader. by uCallHimDrJ0NES · · Score: 5, Insightful

    First rule of leadership: Everything is your fault.

    --
    Cloudiot: A person who does not see offsite storage as a way to lose control over access to his or her own data.
  8. Yes, it's the LOW level employees by gurps_npc · · Score: 5, Insightful
    If upper management is not aware that lower level employees are engaging in a massive fraud against their customers, than that means:

    1) Upper management are morons that have no idea what is going on in their company. It's the equivalent of a farmer claiming he had no idea that his 'organic' corn is actually bio-engineered and covered with Round-Up.

    2) That they personally are directly and legally responsible by failing to manage their employees. The buck stops at the BOSS, not the janitor.

    3) Are also committing the Wage-Theft by not doing their official declared job of MANAGING their employees.

    Claiming ignorance, stupidity, and incompetence is not a valid legal defense.

    --
    excitingthingstodo.blogspot.com
  9. Bullocks. by SvnLyrBrto · · Score: 5, Insightful

    This isn't some little project where one or two rogue engineers can throw a commit into github without oversight. We're talking about a major, multi-million dollar engineering project that spans both software and hardware, goes into a production run of many thousands of vehicles, and is regulated by many governmental bodies across multiple countries.

    At a minimum, you'd need the involvement of:

    The software engineers
    The hardware engineers
    The integration engineers
    The software QA testers
    The hardware QC testers
    The integration testers
    The production engineers
    The production QC testers
    Various compliance managers
    Whoever is submitting the test vehicles to the government testers in each country.
    The managers and supervisors of all of the above

    With that many people involved... and that's probably a conservative list... it's hard to believe that there wasn't some C-level approval or direction. Massive fraud in a major engineering project doesn't bubble up from one rogue employee or two. It's rolled from the top down.

    --
    Imagine all the people...
  10. Link does not go to the article by Geoffrey.landis · · Score: 4, Informative

    The link does not go to the article. Could somebody post the actual link?

    Here are some other sources:
    http://www.newser.com/story/21...
    http://www.theguardian.com/bus...
    http://www.npr.org/sections/th...

    --
    http://www.geoffreylandis.com
  11. More like Large Bogus by SuperKendall · · Score: 5, Insightful

    and the boss promised us a large bonus if we can do it with software

    Now I know this is fanfic because no-where in any real company have I, as a software engineer, been promised a bonus for doing ANYTHING.

    --
    "There is more worth loving than we have strength to love." - Brian Jay Stanley
  12. This just in by easyTree · · Score: 3, Funny

    Hitler's ghost blames holocaust on single soldier.

    Sorry Mike.

  13. What if they all behave that way by rsilvergun · · Score: 3, Interesting

    What then? That's really why this is blowing up so much. Seriously. VW is going to get fined a couple of billion. That's not even chump change. Nobody at that level gives two shits about pride, and the average consumer will forget this same as they forgot Toyota's acceleration problems. What _has_ come out is that _everyone_ was cheating, they were just better at it so that when they got caught there was some doubt and nobody got in real trouble. So what the hell do you do if you're an engineer and this is industry practice?

    It's like a buddy of mine who used to drive truck and followed the rules. He went from company to company and they all promised him he'd never drive over limit. And when he didn't they eventually stopped giving him runs. For all you're talk that's not the way the real world works.

    --
    Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/