Should I Take Toyota's Software Update?

kiehlster writes "I'm a software developer, and I know that most software has bugs, but how much trust can we put in the many lines of code found in our automobiles? I have a 2009 Camry that is involved in both of the recent Toyota recalls. As part of the floor-mat issue, they're offering to install a software update that would cause 'the brake pedal to take precedence over the gas pedal if both were pressed,' or, as their latest notice states, 'would cut power to the engine if both pedals were pressed.' In the computer world, we're all taught to install firmware updates only if there is a real problem because a large percentage of firmware updates actually brick the hardware or cause other unforeseen consequences. On a base of 100 million lines of code, can I really trust a software update to work safely when it is delivered in a three-month development cycle? My driving habits don't cause the floor mat to slide much, so I see the update as overkill. What do you think? If it doesn't void the warranty, should I tell them to skip the update?"

Comments

You're looking at it wrong.

[Anonymous Coward]

You already took the 100 million lines of code when you bought the car.

Now do you want the bug fixes, or would you rather find out what a "fatal exception" means in more physical terms?

Re:You're looking at it wrong.

[Rakshasa Taisab]

Good luck getting any money from Toyota or your insurance company if you _don't_ take that update.

Besides, there's not 100 million lines of code in _that_ particular part, they won't be updating your blinkenlights firmware and such at the same time.

Re:You're looking at it wrong.

[0100010001010011]

It's not 100M lines of handwritten code! Every time this comes up everyone (especially those that work with embedded systems) seem to think that there are a ton of code monkeys locked away coding in C or assembly.

I'd be willing to bet that almost all of it is auto generated. Toyota (and nearly everyone else) uses Matlab & Simulink extensively.
The MathWorks tools help Toyota design for the future (PDF)

Toyota Racing Development Makes Faster and More Efficient Engineering Decisions with MATLAB

A simple PID controler with saturation and limits could easily take up 50 "lines of code".

And it's not like Toyota is Mathworks' sole customer. Boeing, GM, Chrysler, Ford, etc ALL use Mathworks.

Just like nearly everyone that works with CAN uses Vector CANape. Everyone that develops ICE powertrains uses AVL

When you start to get to specialized software like what Matlab, CANape, AVL, etc all do, there aren't a ton of options (and no open source solutions). It's cheaper for all of these companies to buy X product and use it than try to write their own.

Re:You're looking at it wrong.

[je ne sais quoi]

Not to mention that there is a real chance this isn't being caused by floor-mats or sticky pedals at all and that it's the software that's causing this in the first place. My gut is to say that their patch is necessary for the same reason why the phone company uses a program whose job it is to go and find memory that is allocated but not being used and free that memory. It's because the system is so complicated that they don't know what's causing the problem and can't find the answer, so this patch acts as a stop-gap to at least cure the symptom if not the disease.

I think you'd have to be nuts not to install it.

Re:You're looking at it wrong.

[Sir_Lewk]

That's like using the LOC count of a disassembled program written in C to express the size of the original code.

Re:You're looking at it wrong.

[clone53421]

Heh. Yeah, that’s about the same response that I have.

The current firmware has a known bug which randomly transforms your car into a flying brick, with you trapped inside, moving at freeway speeds.

Updating the firmware involves the risk that your car will be transformed into a stationary brick, with you nowhere around, and with your dealer on the hook to get it fixed.

Let me see... how long does the cost vs. benefit analysis take on this one?

Re:You're looking at it wrong.

[schlesinm]

The dealer is doing the firmware update as part of the recall. If they brick your car because the firmware modification goes wrong, then they replace the bricked part. There is no risk on that side. So the big question is do you want a fix for a known bug or do you want to keep the buggy firmware. And as the parent says, if you don't do the upgrade, then if the bug happens to you the insurance company and manufacturer will deny your claim because you refused to fix the bug.

Re:You're looking at it wrong.

[odin84gk]

As a user of these software programs, I can tell you how they are Really used:
PHD Uses matlab and simulink to create their motor control algorithms. They port program to the processor of choice and test their algorithm.
Once their algorithm is proved, the firmware engineer uses that code as a template. They re-write all the code to play nicely with the other required code and to improve efficiency. (WTF? Another Memcopy? GARGH! Stop hogging all of my cycles!)

It is a great program for a rapid prototype and proof-of-concept, but it totally fails on actual implementation. I have been to a few microcontroller workshops where people have told the horror stories about the atrocious code created by these programs. In the end, it is just not production quality code.

Re:You're looking at it wrong.

[TheLink]

Which articles were that?

The one I saw was this:
http://www.caranddriver.com/features/09q4/how_to_deal_with_unintended_acceleration-tech_dept

The speed where brakes+full throttle didn't eventually stop the car was 120mph.

And their conclusion:
http://www.caranddriver.com/news/car/10q1/toyota_recall_scandal_media_circus_and_stupid_drivers-editorial

Re:You're looking at it wrong.

[Zurk]

IT is not THE fix. it is a failsafe for THE fix.
The REAL problem is the reading from the toyota ECM when the two redundant APP (accln pedal position) signal circuits are shorted together (main and sub), From the toyota camry VSRM :
DESCRIPTION
This ETCS (Electronic Throttle Control System) does not use a throttle cable. The Accelerator Pedal Position (APP) sensor is mounted on the accelerator pedal bracket and has 2 sensor circuits: VPA (main) and VPA2 (sub). This sensor is a non-contact type, and uses Hall-effect elements, in order to yield accurate signals, even in extreme driving conditions, such as at high speeds as well as very low speeds. The voltage, which is applied to terminals VPA and VPA2 of the ECM, varies between 0 V and 5 V in proportion to the operating angle of the accelerator pedal (throttle valve). A signal from VPA indicates the actual accelerator pedal opening angle (throttle valve opening angle) and is used for engine control. A signal from VPA2 conveys the status of the VPA circuit and is used to check the APP sensor itself. The ECM monitors the actual accelerator pedal opening angle (throttle valve opening angle) through the signals from VPA and VPA2, and controls the throttle actuator according to these signals.

FAIL-SAFE
The accelerator pedal position sensor has two (main and sub) sensor circuits. If a malfunction occurs in either of the sensor circuits, the ECM detects the abnormal signal voltage difference between the two sensor circuits and switches to limp mode. In limp mode, the functioning circuit is used to calculate the accelerator pedal opening angle to allow the vehicle to continue driving. If both circuits malfunction, the ECM regards the opening angle of the accelerator pedal as being fully closed. In this case, the throttle valve remains closed as if the engine is idling.
If a pass condition is detected and then the ignition switch is turned off, the fail-safe operation stops and the system returns to a normal condition.

VPA and VPA2 are coming from the PCM with .5-1.1v at one of the sensors and 1.2-2.0v at the other when the pedal is at its relaxed position. When there's force at the pedal, one sensor will operate between 2.6-4.5v and the other at 3.4-5.0v.

Toyota specs normal voltage for both the VPA sensors between between .4-4.8v for VPA, and .5-4.8v for VPA2 with a .2v deviation between the 2 sensors. Anything out of those ranges will trigger a DTC

An internal short could occur within one or more of the paths from the circuits leading to the ecm. That could lead to a situation where the computer cannot detect its own failure.Therefore, when the system gets conflicting information, it arbitrarily ignores half the conflicting information. It does not know which of the circuits are lying or if they both are lying and shorted together. different resistance values will lead to arbitrary acceleration. Having the brake override it is a stopgap, but ixing the real problem (perhaps with a third circuit in voting mode which will require replacing the entire circuit path) is the REAL FIX. I suspect 2012 and onwards toyotas would have a third path and faraday cage/denso replacement for the magnet assembly in the plastic accelerator pedal (which is another problem with EMI which might lead to acceleration) which i am not going to go into here.

So, YES OP you should definitely install the update. Its the only thing standing between you and death if both the APP circuits short.

Re:You're looking at it wrong.

[Andy Dodd]

My background is as an RF engineer, and I have a reasonable familiarity with EMI engineering.

The utter fucking cluelessness of that article scares me.

"Professor Liu, the story says, compares it to the problem with the jamming of signals on military aircraft.

"The problem is, the expertise for preventing signal jamming rests in the Department of Defense, not the automakers or their suppliers,' Professor Liu says. "
There's a MASSIVE difference between trying to prevent jamming of communications/radar signals, and basic EMI protection engineering of wired electronic circuits. There is PLENTY of experience with the latter in the civilian world, especially within the automotive industry.

Yes, cell phones can cause EMI problems with unshielded equipment, especially GSM phones. The critical systems in a vehicle are without any doubt *shielded*. More details on that later...

Satellite radios are RECEIVERS. (With the exception of satphones - these are incredibly rare.) They can be jammed, but you have to SERIOUSLY fuck up for one of them to interfere with something else. Same for GPS receivers. The most likely way for either of these systems to affect a car negatively is for them to short out and pull excessive current from their power supply. That's what fuses are for.

Large restaurant microwaves are subject to the same restrictions from the FCC as home microwaves. Yeah they can leak a little and they'll jam 2.4 GHz communications, but you could most likely take the magnetron from a microwave oven, point it at a car, and no adverse effects to critical systems would happen.

Why? Because the ignition system within a car is typically the #1 source of interference to anything in or near a car. A malfunctioning ignition system (old spark plug wires, loose spark plug wire connections) is tantamount to a high power spark gap transmitter. Automotive engineers have been dealing with internally generated EMI since the beginning of their industry.

Re:You're looking at it wrong.

[0100010001010011]

Ok. Case in point, here is a VERY simple switch block. (And this could really be all that they did)

Brake_Override.jpg

If brake is 1, then 0 gets sent to the throttle, otherwise what ever the throttle is gets sent to the throttle.

How many lines of code would you guess that is?
157. (including blank lines between functions).

Want to wager how many the .h file has?

901.

For that little model right there, there were almost 1000 lines of code. Now do you see how you could easily get 100M?

*This is also quick and dirty, I didn't turn on any optimizations it's just the default C generated code to make a .exe (I didn't target any specific embedded device).

**Now in real production these would pull from sensors and it'd probably use a few more lines of code. (You have to read from the A/D, etc)

Re:You're looking at it wrong.

[DerekLyons]

So he's using it wrong because he optimizes it and actually evaluates the running code, and you're using it correctly because you treat it as a black box?

Interesting.

Re:You're looking at it wrong.

[cgenman]

I would add that the "floor mat" excuse always sounded like BS to me. I'm guessing there is a firmware bug in there somewhere that they can't find that just registers the gas pedal as down. They'd never admit to that, as it would reduce the public perception of security of drive-by-wire systems, and might introduce expensive public testing procedures.

In that case, your only chance is the brake overriding the gas (a process which should have been true from the beginning anyway). Of course, it might be something else and you might still be screwed... unknown computer bugs are like that.

Re:You're looking at it wrong.

[RotsiserMho]

Or the first guy is using it wrong and taking the chance of introducing even MORE bugs (more cooks in the kitchen) while the second guy is relying on code that has been tested time and time again, not only by the Mathworks, but by all of their customers as well. Tell me, when writing code for Linux do you re-evaluate every line of the kernel or treat it as a black box? One of our largest customers (a Fortune 100 heavy equipment manufacturer) relies on generated code to control their engines. And these are big engines. The Mathworks produces very solid code allowing developers to create control systems very quickly that are time-tested to be reliable. That being said, that doesn't mean Toyota simply didn't connect the blocks wrong in this case. A human is still responsible for the logic.

Umm... yes

[Anonymous Coward]

Unpatched PCs are bad enough. If I can't go outside because of morons with unpatched cars, I will be very unhappy.

Take the update

[FrYGuY101]

If it bricks, the Dealer's going to be the one who has to replace it. As far as I look at it, it's zero risk, financially.

Safety wise, it fixes a known bug.

Take the update.

Re:Take the update

[Goobermunch]

A bug that you know about. If, by chance, you find yourself in an accident, and get sued, I doubt a jury is going to look kindly on the "I passed up on the fix for the known bug because I thought it might brick my car" defense. If you pass on the deal, you are essentially taking full responsibility for Toyota's bad code.

That's not a good choice.

--AC

Re:yes

[Anonymous Coward]

Uh - if the dealership "bricks" your car by applying the update they will fix it for free. This question is just plain stupid - get the damn update. If something ever happens and you crash your car the first thing they will say is that you declined to apply their update and so they are not liable.

I wouldn't do it

[BadAnalogyGuy]

There's the chance that the update may turn off any jailbreaks you've already got working. Worst case scenario is that it detects a jailbreak and bricks your car, like you said.

I'd stick with the white hat hackers who are providing jailbreaking instructions and forgo any manufacturer updates.

The worst that can happen is that your car becomes susceptible to the sudden acceleration "problem" and you lose control and wipe out a family or farmer's market. But you're inside the car so you'll be fine.

Plus, you'd have to go down to the dealership and they're going to ask you if you've had any problems and a huge rigmarole just to end up with essentially the same performance you've had all along.

Too many risks and too few benefits. I'd say no.

Get the Flash

[nicholasjay]

There's a lot of cars that have the 'brake takes precedence' feature. The only real reason to not have such a feature is because of trail-braking or hell-toe shifting. Both are racing/performance driving techniques you won't be doing in your Camry. Plus, it is a pure software feature in that if it detects you braking, it will cut throttle. So there's no big issue there.

Also, cars have their computers updated all the time, and it has never been a big deal in the past. The Nissan GTR was the last example that made the news (to cut down on the RPM the launch control used). But really, cars are reflashed all the time. Its not a big deal.

Re:huh?

[wjsteele]

Agreed... they've already had problems with it and NOT ACCEPTING the fix for it sounds kind of stupid to me. On second thought, maybe the GP should not accept the fix and let Darwin do his magic. Especially since the logic is so simple... if I'm pressing on the brake, don't give the engine gas. Seems like no brainer to me... I mean the fix, not the GP... on second thought, they both do.

Bill