Should I Take Toyota's Software Update?

kiehlster writes "I'm a software developer, and I know that most software has bugs, but how much trust can we put in the many lines of code found in our automobiles? I have a 2009 Camry that is involved in both of the recent Toyota recalls. As part of the floor-mat issue, they're offering to install a software update that would cause 'the brake pedal to take precedence over the gas pedal if both were pressed,' or, as their latest notice states, 'would cut power to the engine if both pedals were pressed.' In the computer world, we're all taught to install firmware updates only if there is a real problem because a large percentage of firmware updates actually brick the hardware or cause other unforeseen consequences. On a base of 100 million lines of code, can I really trust a software update to work safely when it is delivered in a three-month development cycle? My driving habits don't cause the floor mat to slide much, so I see the update as overkill. What do you think? If it doesn't void the warranty, should I tell them to skip the update?"

Comments

You're looking at it wrong.

[Anonymous Coward]

You already took the 100 million lines of code when you bought the car.

Now do you want the bug fixes, or would you rather find out what a "fatal exception" means in more physical terms?

Re:You're looking at it wrong.

[Rakshasa Taisab]

Good luck getting any money from Toyota or your insurance company if you _don't_ take that update.

Besides, there's not 100 million lines of code in _that_ particular part, they won't be updating your blinkenlights firmware and such at the same time.

Re:You're looking at it wrong.

[0100010001010011]

It's not 100M lines of handwritten code! Every time this comes up everyone (especially those that work with embedded systems) seem to think that there are a ton of code monkeys locked away coding in C or assembly.

I'd be willing to bet that almost all of it is auto generated. Toyota (and nearly everyone else) uses Matlab & Simulink extensively.
The MathWorks tools help Toyota design for the future (PDF)

Toyota Racing Development Makes Faster and More Efficient Engineering Decisions with MATLAB

A simple PID controler with saturation and limits could easily take up 50 "lines of code".

And it's not like Toyota is Mathworks' sole customer. Boeing, GM, Chrysler, Ford, etc ALL use Mathworks.

Just like nearly everyone that works with CAN uses Vector CANape. Everyone that develops ICE powertrains uses AVL

When you start to get to specialized software like what Matlab, CANape, AVL, etc all do, there aren't a ton of options (and no open source solutions). It's cheaper for all of these companies to buy X product and use it than try to write their own.

Re:You're looking at it wrong.

[je ne sais quoi]

Not to mention that there is a real chance this isn't being caused by floor-mats or sticky pedals at all and that it's the software that's causing this in the first place. My gut is to say that their patch is necessary for the same reason why the phone company uses a program whose job it is to go and find memory that is allocated but not being used and free that memory. It's because the system is so complicated that they don't know what's causing the problem and can't find the answer, so this patch acts as a stop-gap to at least cure the symptom if not the disease.

I think you'd have to be nuts not to install it.

Re:You're looking at it wrong.

[Sir_Lewk]

That's like using the LOC count of a disassembled program written in C to express the size of the original code.

Re:You're looking at it wrong.

[clone53421]

Heh. Yeah, that’s about the same response that I have.

The current firmware has a known bug which randomly transforms your car into a flying brick, with you trapped inside, moving at freeway speeds.

Updating the firmware involves the risk that your car will be transformed into a stationary brick, with you nowhere around, and with your dealer on the hook to get it fixed.

Let me see... how long does the cost vs. benefit analysis take on this one?

Re:You're looking at it wrong.

[urulokion]

I doubt the primary motivation is because of a suspected software problem. I'd say the primary motivation is because Toyota is the one (or one of the few) car manufacture that didn't have a brake-override feature in their fly-by-wire vehicles. After all of the publicity about the raw away cars, they are pulling out the stops to prevent it from getting worse.

I think it was Car and Driver who did a test of vehicles which had fly-by-wire throttle systems to see how they handled under runaway conditions. They basically took the cars up to certain speeds (20, 40 and 60 MPH IIRC), kept the throttle depressed, and then tried to stop the car with brakes and emergency breaks. Every vehicle with the brake override system, the engines immediately went down to idle power when the brakes where pressed even with the thottle held down. It was very easy to bring the vehicle to a controlled stop.

The Toyotas w/o the brake override system could be stopped if you were at slow speeds with a lot of effort on the brakes and emergency brake. At higher speeds, the breaks where not enough to stop the vehicle with only the brakes. They also tried turning the vehicles off which would stop the vehicle, but the driver had to manhandle the vehicle w/o benefit of power steering and power brakes.

Side note: The Toyota Prius has a surprising amount of power at full ouput. That's when the gas engine is driving the wheels, teh eletric drive motor is drawing off teh traction battery to drive the wheels, and the gas engine is driving a secondary motor/generator to creating electricity which is feed to the eletric drive motor. The secondary motor/generator is normally used to recharge the traction battery when the car is operating in usual conditions.

I was doing 65-75 MPH up the foothills in Arizona and Southern California. I was outdoing a lot of other vehicles with power engines. My cruise control kept at the set speed and didn't slow down at all. Unfortunately the Prius can only maintain that kind of output as the traction battery charge lasts. And the gas milage really sucks in that mode.

Re:You're looking at it wrong.

[schlesinm]

The dealer is doing the firmware update as part of the recall. If they brick your car because the firmware modification goes wrong, then they replace the bricked part. There is no risk on that side. So the big question is do you want a fix for a known bug or do you want to keep the buggy firmware. And as the parent says, if you don't do the upgrade, then if the bug happens to you the insurance company and manufacturer will deny your claim because you refused to fix the bug.

Re:You're looking at it wrong.

[odin84gk]

As a user of these software programs, I can tell you how they are Really used:
PHD Uses matlab and simulink to create their motor control algorithms. They port program to the processor of choice and test their algorithm.
Once their algorithm is proved, the firmware engineer uses that code as a template. They re-write all the code to play nicely with the other required code and to improve efficiency. (WTF? Another Memcopy? GARGH! Stop hogging all of my cycles!)

It is a great program for a rapid prototype and proof-of-concept, but it totally fails on actual implementation. I have been to a few microcontroller workshops where people have told the horror stories about the atrocious code created by these programs. In the end, it is just not production quality code.

Re:You're looking at it wrong.

[TheLink]

Which articles were that?

The one I saw was this:
http://www.caranddriver.com/features/09q4/how_to_deal_with_unintended_acceleration-tech_dept

The speed where brakes+full throttle didn't eventually stop the car was 120mph.

And their conclusion:
http://www.caranddriver.com/news/car/10q1/toyota_recall_scandal_media_circus_and_stupid_drivers-editorial

Re:You're looking at it wrong.

[0100010001010011]

Then you're using it wrong.

I work for a rather large corporation that uses Simulink for all of our stuff. Nothing gets re-written. The stuff that goes into production is stuff that IS assembled by the electronics group.

Other groups that design the control algorithms do use XPC boxes to create strategies quickly. Once this is done a software specification is written and given to the group that actually makes the model 'their way' (fixed point, design standards, naming conventions, etc). This gets compiled and put into production ECMs that customers use.

It's really amazing how settings and maps get pulled from different databases and merged together

Re:You're looking at it wrong.

[Zurk]

IT is not THE fix. it is a failsafe for THE fix.
The REAL problem is the reading from the toyota ECM when the two redundant APP (accln pedal position) signal circuits are shorted together (main and sub), From the toyota camry VSRM :
DESCRIPTION
This ETCS (Electronic Throttle Control System) does not use a throttle cable. The Accelerator Pedal Position (APP) sensor is mounted on the accelerator pedal bracket and has 2 sensor circuits: VPA (main) and VPA2 (sub). This sensor is a non-contact type, and uses Hall-effect elements, in order to yield accurate signals, even in extreme driving conditions, such as at high speeds as well as very low speeds. The voltage, which is applied to terminals VPA and VPA2 of the ECM, varies between 0 V and 5 V in proportion to the operating angle of the accelerator pedal (throttle valve). A signal from VPA indicates the actual accelerator pedal opening angle (throttle valve opening angle) and is used for engine control. A signal from VPA2 conveys the status of the VPA circuit and is used to check the APP sensor itself. The ECM monitors the actual accelerator pedal opening angle (throttle valve opening angle) through the signals from VPA and VPA2, and controls the throttle actuator according to these signals.

FAIL-SAFE
The accelerator pedal position sensor has two (main and sub) sensor circuits. If a malfunction occurs in either of the sensor circuits, the ECM detects the abnormal signal voltage difference between the two sensor circuits and switches to limp mode. In limp mode, the functioning circuit is used to calculate the accelerator pedal opening angle to allow the vehicle to continue driving. If both circuits malfunction, the ECM regards the opening angle of the accelerator pedal as being fully closed. In this case, the throttle valve remains closed as if the engine is idling.
If a pass condition is detected and then the ignition switch is turned off, the fail-safe operation stops and the system returns to a normal condition.

VPA and VPA2 are coming from the PCM with .5-1.1v at one of the sensors and 1.2-2.0v at the other when the pedal is at its relaxed position. When there's force at the pedal, one sensor will operate between 2.6-4.5v and the other at 3.4-5.0v.

Toyota specs normal voltage for both the VPA sensors between between .4-4.8v for VPA, and .5-4.8v for VPA2 with a .2v deviation between the 2 sensors. Anything out of those ranges will trigger a DTC

An internal short could occur within one or more of the paths from the circuits leading to the ecm. That could lead to a situation where the computer cannot detect its own failure.Therefore, when the system gets conflicting information, it arbitrarily ignores half the conflicting information. It does not know which of the circuits are lying or if they both are lying and shorted together. different resistance values will lead to arbitrary acceleration. Having the brake override it is a stopgap, but ixing the real problem (perhaps with a third circuit in voting mode which will require replacing the entire circuit path) is the REAL FIX. I suspect 2012 and onwards toyotas would have a third path and faraday cage/denso replacement for the magnet assembly in the plastic accelerator pedal (which is another problem with EMI which might lead to acceleration) which i am not going to go into here.

So, YES OP you should definitely install the update. Its the only thing standing between you and death if both the APP circuits short.

Re:You're looking at it wrong.

[frog_strat]

I was on a medical device project using generated code. After three years, management directed us to dump the generated code and hand code it. The two reasons were 1) known bad code the (widely used) tool was generating 2) Code generator company would not certify the generated code, regardless of what we were willing to pay. Required for medical.

Re:You're looking at it wrong.

[Andy Dodd]

My background is as an RF engineer, and I have a reasonable familiarity with EMI engineering.

The utter fucking cluelessness of that article scares me.

"Professor Liu, the story says, compares it to the problem with the jamming of signals on military aircraft.

"The problem is, the expertise for preventing signal jamming rests in the Department of Defense, not the automakers or their suppliers,' Professor Liu says. "
There's a MASSIVE difference between trying to prevent jamming of communications/radar signals, and basic EMI protection engineering of wired electronic circuits. There is PLENTY of experience with the latter in the civilian world, especially within the automotive industry.

Yes, cell phones can cause EMI problems with unshielded equipment, especially GSM phones. The critical systems in a vehicle are without any doubt *shielded*. More details on that later...

Satellite radios are RECEIVERS. (With the exception of satphones - these are incredibly rare.) They can be jammed, but you have to SERIOUSLY fuck up for one of them to interfere with something else. Same for GPS receivers. The most likely way for either of these systems to affect a car negatively is for them to short out and pull excessive current from their power supply. That's what fuses are for.

Large restaurant microwaves are subject to the same restrictions from the FCC as home microwaves. Yeah they can leak a little and they'll jam 2.4 GHz communications, but you could most likely take the magnetron from a microwave oven, point it at a car, and no adverse effects to critical systems would happen.

Why? Because the ignition system within a car is typically the #1 source of interference to anything in or near a car. A malfunctioning ignition system (old spark plug wires, loose spark plug wire connections) is tantamount to a high power spark gap transmitter. Automotive engineers have been dealing with internally generated EMI since the beginning of their industry.

Re:You're looking at it wrong.

[0100010001010011]

Ok. Case in point, here is a VERY simple switch block. (And this could really be all that they did)

Brake_Override.jpg

If brake is 1, then 0 gets sent to the throttle, otherwise what ever the throttle is gets sent to the throttle.

How many lines of code would you guess that is?
157. (including blank lines between functions).

Want to wager how many the .h file has?

901.

For that little model right there, there were almost 1000 lines of code. Now do you see how you could easily get 100M?

*This is also quick and dirty, I didn't turn on any optimizations it's just the default C generated code to make a .exe (I didn't target any specific embedded device).

**Now in real production these would pull from sensors and it'd probably use a few more lines of code. (You have to read from the A/D, etc)

Re:You're looking at it wrong.

[DerekLyons]

So he's using it wrong because he optimizes it and actually evaluates the running code, and you're using it correctly because you treat it as a black box?

Interesting.

Re:You're looking at it wrong.

[cgenman]

I would add that the "floor mat" excuse always sounded like BS to me. I'm guessing there is a firmware bug in there somewhere that they can't find that just registers the gas pedal as down. They'd never admit to that, as it would reduce the public perception of security of drive-by-wire systems, and might introduce expensive public testing procedures.

In that case, your only chance is the brake overriding the gas (a process which should have been true from the beginning anyway). Of course, it might be something else and you might still be screwed... unknown computer bugs are like that.

Re:You're looking at it wrong.

[ckaminski]

Which is why I don't like push-button ignition. If my car ever goes into hyperdrive because of a stuck throttle, I take comfort in knowing I still have a kill switch, and I grew up driving tractors and cars without power steering or power-assist braking, so I can cope.

How can I trust that that push-button ignition will still shut off the car? I know it's conceivable that even a key-start ignition might turn all ignition control over to an ECM, but who's done that?

Re:You're looking at it wrong.

[RotsiserMho]

Or the first guy is using it wrong and taking the chance of introducing even MORE bugs (more cooks in the kitchen) while the second guy is relying on code that has been tested time and time again, not only by the Mathworks, but by all of their customers as well. Tell me, when writing code for Linux do you re-evaluate every line of the kernel or treat it as a black box? One of our largest customers (a Fortune 100 heavy equipment manufacturer) relies on generated code to control their engines. And these are big engines. The Mathworks produces very solid code allowing developers to create control systems very quickly that are time-tested to be reliable. That being said, that doesn't mean Toyota simply didn't connect the blocks wrong in this case. A human is still responsible for the logic.

Re:You're looking at it wrong.

[nacturation]

Push-button ignition can be turned off by holding down the button (kind of like with a computer)...

... and waiting for the software interrupt to get picked up by the CPU, which may be in a hung state.

Re:You're looking at it wrong.

[toastar]

Just push the power button for 5 seconds.

Yah Know.... I never really liked when computers switched to this method with the ATX revolution, Sometimes you still have to reach around and pull the plug. Sometimes it can take a minute or two.

I'd hate for this to happen in a life or death scenario. As mentioned above a hard off ala old AT cases just seams safer.

Re:You're looking at it wrong.

[sorak]

Push-button ignition can be turned off by holding down the button (kind of like with a computer)...

... and waiting for the software interrupt to get picked up by the CPU, which may be in a hung state.

Not to mention that it's hard to hold a button down for three seconds while you're weaving in and out of traffic and urinating on yourself.

huh?

[pele]

Are you for real?

Re:huh?

[wjsteele]

Agreed... they've already had problems with it and NOT ACCEPTING the fix for it sounds kind of stupid to me. On second thought, maybe the GP should not accept the fix and let Darwin do his magic. Especially since the logic is so simple... if I'm pressing on the brake, don't give the engine gas. Seems like no brainer to me... I mean the fix, not the GP... on second thought, they both do.

Bill

yes

[samyem]

yes

Re:yes

[Anonymous Coward]

Uh - if the dealership "bricks" your car by applying the update they will fix it for free. This question is just plain stupid - get the damn update. If something ever happens and you crash your car the first thing they will say is that you declined to apply their update and so they are not liable.

If it bricks, it's their fault.

[rotide]

First, this is about your safety.

Second, if the update bricks your car, that would be Toyota's fault, not yours and I'm pretty sure they would resolve the issue for you free of charge.

Or, you can keep driving a potentially unsafe vehicle on "firmware update" principles.

Umm... yes

[Anonymous Coward]

Unpatched PCs are bad enough. If I can't go outside because of morons with unpatched cars, I will be very unhappy.

Take the update

[FrYGuY101]

If it bricks, the Dealer's going to be the one who has to replace it. As far as I look at it, it's zero risk, financially.

Safety wise, it fixes a known bug.

Take the update.

Re:Take the update

[Goobermunch]

A bug that you know about. If, by chance, you find yourself in an accident, and get sued, I doubt a jury is going to look kindly on the "I passed up on the fix for the known bug because I thought it might brick my car" defense. If you pass on the deal, you are essentially taking full responsibility for Toyota's bad code.

That's not a good choice.

--AC

Are you kidding?

[Spazmania]

Take the upgrade. Shipping firmware always has bugs. Always. As a system administrator, the first thing I do out of the box is download and install the current firmware while it's still under warranty. And if they brick your computer they'll replace it.

Safety First

[Linker3000]

Yes, but make sure you drive the Toyota round a large sandbox for a few days first...maybe you live near a sandy beach or golf course with large bunkers. At a pinch, do your kids have a playpit in the garden? Cat litter tray?

I wouldn't do it

[BadAnalogyGuy]

There's the chance that the update may turn off any jailbreaks you've already got working. Worst case scenario is that it detects a jailbreak and bricks your car, like you said.

I'd stick with the white hat hackers who are providing jailbreaking instructions and forgo any manufacturer updates.

The worst that can happen is that your car becomes susceptible to the sudden acceleration "problem" and you lose control and wipe out a family or farmer's market. But you're inside the car so you'll be fine.

Plus, you'd have to go down to the dealership and they're going to ask you if you've had any problems and a huge rigmarole just to end up with essentially the same performance you've had all along.

Too many risks and too few benefits. I'd say no.

Get the Flash

[nicholasjay]

There's a lot of cars that have the 'brake takes precedence' feature. The only real reason to not have such a feature is because of trail-braking or hell-toe shifting. Both are racing/performance driving techniques you won't be doing in your Camry. Plus, it is a pure software feature in that if it detects you braking, it will cut throttle. So there's no big issue there.

Also, cars have their computers updated all the time, and it has never been a big deal in the past. The Nissan GTR was the last example that made the news (to cut down on the RPM the launch control used). But really, cars are reflashed all the time. Its not a big deal.

Apply the update

[Cassini2]

Many other manufacturers have already added a similar piece of code. It really doesn't take to long to debug an interlock. Your primary failure mode will be: if the brake pressed switch fails (ie: the tail lights are stuck on), then the car won't run.

Every interlock has a strong tendency to fail into the safe state. Conversely, omitting interlocks tends to result in fail-dangerous failures, which is what Toyota is experiencing.

Seriously?

[clone53421]

Take the update.

My driving habits don't cause the floor mat to slide much, so I see the update as overkill.

Perhaps, but didn’t I read about some people who died in a Toyota, presumably from this exact bug, whose floor mat was found secure in their trunk, exactly where Toyota recommended them to put it when they thought the floor mats were causing the accelerator bug?

Absolutely

[onyxruby]

Think of this a few different ways. First from a liability standpoint, you are considering actively refusing a fix for a known bug that has killed people. If you ever sell your car and it can be proved you actively refused this you could be on the hook both civilly and criminally. Second from a liability standpoint, Toyota is now assuming liability for this, if they brick your car, they are liable for fixing it. Third, this is a known bug that has killed people, are you bloody nuts? This is not a software bug that results in a software crash, this is a software bug that results in a real world crash!

Jane, you ignorant slut...

[HotNeedleOfInquiry]

In the computer world, we're all taught to install firmware updates only if there is a real problem because a large percentage of firmware updates actually brick the hardware or cause other unforeseen consequences.

Nobody taught you that. You pulled it out of your ass so you'd sound officious and get a post on /.

The vast majority of firmware updates work, fix problems and don't brick devices. Much more of this shit that gets by as posts and I'll be begging for Jon Katz to come back.

Re:Jane, you ignorant slut...

[SuiteSisterMary]

I believe, truely and honestly, that the submitter thinks that he's expected to go to www.toyota.com, click on 'support,' 'downloads,' 'firmware,' 'by make and model,' and download a binary file which goes onto a USB key.

I believe that the submitter then thinks there will be instructions like 'pop the cover on the fuse panel, and insert the USB key containing the firmware upgrade in the USB slot. Start the car while holding both the 'rear window defroster' and 'left turn signal' down. The car will start in firmware upgrade mode and automatically start upgrading the firmware. DO NOT POWER OFF THE CAR DURING THIS TIME.'

If you don't

[cmiller173]

If you don't take the patch and later have the problem you will likely have lost the ability to sue if necessary. Also, if you live in a state with the concept of "contributory negligence" in it's laws you could be found partially or fully at fault for any accidents that would have been prevented by the patch. Eventually insurance companies are going to realize that they could deny claims in accidents if the driver's car is not fully patched. So yes, take the patch

I will be getting that firmware update

[urulokion]

I have an '09 Prius. And I'll be getting that firmware update. It's a feature they should have included in the first place. It's not the best implementation of the brake override I'd like. What I'd really like to have an electrical circuit connection between the brake pedal and the throttle fly-by-wire assembly. When the circuit is tripped, the throttle position output of the assembly drops to 0 regardless of actual pedal position or sensor position. But that would require new hardware.

I'm getting the update because if the engine does start runaway acceleration, the brakes aren't enough to overcome the hybrid system's output. I know the right thing to do would be to put the car into neutral and get it safely off the road. But I don't react well to stressful situations.

He is looking at it wrong...

[Oxford_Comma_Lover]

> ''the brake pedal to take precedence over the gas pedal if both were pressed' or, as their latest notice states, 'would cut power to the engine if both pedals were pressed.'

Hint: this is a feature, not a bug. And even if you're reviewing very closely, it's not something that it takes three months to avoid messing up. if(X&&Y) Z=Y;

When the two pedals work at the same time, it can result in pretty horrible accidents. Unless your driving style uses both pedals at the same time in a way that increases your safety (in which case you're James Bond and you don't ask slashdot questions), just take the update.

Re:He is looking at it wrong...

[uglyduckling]

You should never roll back at all. I'm in the UK so I always drive manual/stick except for a few months when I had my parents' automatic. On steep inclines you should always use the handbrake to move off; on more shallow inclines you can quickly move the right foot from the brake to the gas whilst slightly moving the clutch up with the left foot to get the clutch to bite. Rolling back a foot would fail a driving test here, and could get you a ticket if the police spot you doing it (although pretty unlikely).

Toyotaphobia getting out of hand

[guanxi]

I think the anti-Toyota mania is getting a little out of hand. The problem caused 34 deaths in 10 years. Given the tens (hundreds?) of millions of Toyotas on the road, it's actually not a big deal. It's an unimaginable tragedy to the people and families that died, and it should be fixed. But as a public safety issue, more people died of lightening strikes and bee stings during that period. Heart disease kills over 1,000 Americans per day. Let's keep it in perspective.

Now we don't trust their firmware updates? I think their safety record is pretty good. You're driving their car at death-defying speeds, aren't you?

The concept of a firmware update for your car is pretty interesting, though.

Re:Their new slogan

[Sperbels]

Where was the Spanish Inquisition errr... Congress when Ford had to recall 4.5 million cars a few months ago due to their cruise control causing fires?

Agreed. This has the feel of a smear campaign to put GM back on top.

Rhonda Smith's story smells fishy

[sjbe]

Rhonda Smith's story of six miles of interstate terror, as her Lexus suddenly zoomed to 100 miles per hour, will set the mood Tuesday for the first congressional hearing on Toyota's acceleration problems.

Yes and if you read more about it you'll find several interesting bits of info. One is that upon inspection there was no evidence that the brakes had been applied, including the MECHANICAL emergency brake. She also claimed under oath that she had complained about the problem to Toyota but the only record Toyota has is for an oil change. She also sold the car to a family member (not something you'd think she'd do if it really were unsafe) and according the the Wall Street Journal the car is still on the road.

Frankly I think there are a lot of people making up stories hoping to get money in a lawsuit, much the same way people made up stories about Audi a few decades ago. Yes, there appear to be some actual problems but there are a lot of liars out there too.