Toyota's Killer Firmware

New submitter Smerta writes "On Thursday, a jury verdict found Toyota's ECU firmware defective, holding it responsible for a crash in which a passenger was killed and the driver injured. What's significant about this is that it's the first time a jury heard about software defects uncovered by a plaintiff's expert witnesses. A summary of the defects discussed at trial is interesting reading, as well the transcript of court testimony. 'Although Toyota had performed a stack analysis, Barr concluded the automaker had completely botched it. Toyota missed some of the calls made via pointer, missed stack usage by library and assembly functions (about 350 in total), and missed RTOS use during task switching. They also failed to perform run-time stack monitoring.' Anyone wonder what the impact will be on self-driving cars?"

Comments

Technology is hard and dangerous

[i kan reed]

I'm convinced. I'll give up my career as a computer programmer now, and go use my bare hands for subsistence farming now. Sorry, I was wrong.

Re:Technology is hard and dangerous

[neoritter]

Or we could present this as the new Therac-25 and learn from it. :)

Re:Technology is hard and dangerous

[vux984]

Realistically, you are quite a bit more likely to die in your classic car than you are in a new car despite issues like this.

The new car brakes better, handles better, is an order of magnitude safer in a collision thanks to the crumple zones, airbags, and modern collision testing requirements. It also uses less fuel, and pollutes less.

I like classics too, but I don't have any illusions that they are generally safer or more reliable. I will give you that they are usually easier to fix (assuming they aren't so classic that parts are a problem) but that doesn't make them safer -- and safety was the underlying catalyst for this discussion.

Re:Technology is hard and dangerous

[es330td]

The problem with "a new car" is that some of the functionality has been taken away from the driver. In a classic car, if I put it in neutral, the gears disengage, especially if it is a stick. I may blow the engine if I push on the clutch and the throttle is stuck but power will be disconnected from the drive wheels. If I turn the key counter clockwise, the car WILL shut off. In a push button start, drive by wire car the driver uses physical inputs to tell the computer to do something and then the computer does it. If due to a software glitch it suddenly decides to max the throttle there isn't much I can do as the driver to stop it, at least not in the very limited time I have before I collide with another car or a wall. It isn't the probability of collision with which I have a problem, but the fact that significant parts of the control of a two ton machine powered by incendiary fuel are put under the control of a computer program.

Re:Technology is hard and dangerous

[SethJohnson]

The metal is so much thicker on those old cars, we had to use a sledge hammer instead of a normal body work hammer to take the dent back out

I apologize if I'm stating the obvious here...

Most older products were over-built for durability because there were not methodologies for engineering minimum material for the required applications. Cars and other things were built with thicknesses of material that were tested and known to work. To reduce that thickness risked approaching an unknown threshold for failure. Trial-and-error was used where budgets allowed to reduce material, but this was an expensive process and in most cases the manufacturer chose to overbuild.

In more recent years, computer modeling has enabled engineers to load test structural designs so that the product can be built with the minimum amount of material required to satisfy the desired application. This benefits the producer, the consumer, and the scrap yard, while delivering overall efficiency.

Re:Technology is hard and dangerous

[TapeCutter]

A big red button on the dash marked "emergency stop". As I said elsewhere I've experienced a jammed mechanical throttle on a Honda 750 motorbike. Because I had a clutch the incident was no danger to anyone or anything except the engine, which screamed it's guts out before I turned it off.

Re:Technology is hard and dangerous

[Frobnicator]

Obviously it can fail, but it's a soft fail. The engine won't run, or more likely won't run well. Sudden acceleration or unstoppable engine though? Forget it. With the throttle plate closed there's no way you can get any more than the power produced at idle, no matter what the ECU does.

That is exactly the thing that makes this jury verdict so suspicious.

The driver was 76 years old at the time. This crash was subject to an NTSB investigation, and investigators found no evidence that it was a software fault or a hardware fault. The crash recorder says the driver pushed the accelerator and was not pushing the brakes, and then the car was hit.

And most interestingly from TFA is the last line. Ten of the 12 jury members said they wanted to punish Toyota.

If he was pushing on the brakes he could have probably overcome most of the force of a sudden accidental acceleration. If he had more time there were other options like shifting to neutral, but he was approaching an intersection.

When I look at it, an older driver and vehicle recording systems saying the accelerator was pressed and the brakes were not, investigators finding no evidence to support the claim of a software failure, and then the jury stating they want to punish Toyota, I don't see this as a good verdict.

Re:Technology is hard and dangerous

[MachDelta]

In a "serious accident", I'd wager my old Chrysler New Yorker against your crumple-zones any day of the week.

You'd lose that bet.
And likely only once.

http://www.youtube.com/watch?v=xtxd27jlZ_g

Not only would I experience far lower acceleration forces than you

No, you'd be experiencing far greater acceleration forces, as if no portion of the car gives way and soaks up kinetic energy, a greater portion of it will be transferred to anything not bolted securely to the frame (eg: you).

I won't end up crumpled in my car's own crumple zone.

The cabin is under no circumstances a crumple zone. Engine and trunk compartments make great crumple zones. The cabin should be a vehicle's Waterloo.

Self-driving cars will come with an EULA

[dclozier]

The owner of a self-driving car will have had to accepted the EULA and accepted not to hold the manufacturer liable for sofware defects. (half joking but I wouldn't rule it out)

Re:Self-driving cars will come with an EULA

[Anonymous Coward]

Won't do any good. I can agree to a hold-harmless provision (and, despite the language of the EULA, such clauses are not actually universal). What I cannot do, is agree to it for someone else. You'd better believe a pedestrian hit by my self-driving car can sue the living daylights out of them. Heck, as previously mentioned, depending on what the particular problem is, *I* can still sue them.

Re:"Impact on self-driving cars?" - None

[neoritter]

It might change the programming language they decide to use though. Pick a language that's more stable at run-time like Ada (missile programming) etc.

' Anyone wonder what the impact will be?

[freakingme]

Sure, they will be more safe. Just like in the aviation industry, where each incident/crash is investigated meticulously, and flying has become safer ever since 1903. With non-selfdriving cars 99% of the incidents were caused by human error. Now no more, so we can fix it!

Relevant paragraph

[michaelmalak]

2nd link, 5th paragraph:

In a nutshell, the team led by Barr Group found what the NASA team sought but couldn’t find: “a systematic software malfunction in the Main CPU that opens the throttle without operator action and continues to properly control fuel injection and ignition” that is not reliably detected by any fail-safe. To be clear, NASA never concluded software wasn’t at least one of the causes of Toyota’s high complaint rate for unintended acceleration; they just said they weren’t able to find the specific software defect(s) that caused unintended acceleration. We did.

The impact on self-driving cars? Documentation.

[wjcofkc]

Anyone wonder what the impact will be on self-driving cars?

A longer chapter on debugging in the first edition of "Programming Self-Driving Cars: The Missing Manual."

Re:It is about time!!!

[c-A-d]

Any engineering project requires that the engineers have to answer for what they've done. The mantra is, "As an engineer, if you fuckup, someone dies." Every engineer, regardless of discipline, needs to understand this and if they don't, should consider going into Liberal Arts or something equally useless where the worst they can do is fuck up my food or drink order.

Re:wtf

[geekoid]

Are you sure you are a software engineer, and not some programmer with delusions of grandeur?
 

Re:The Toyota Way

[ramk13]

Did you read TFA?

In a nutshell, the team led by Barr Group found what the NASA team sought but couldn’t find: “a systematic software malfunction in the Main CPU that opens the throttle without operator action and continues to properly control fuel injection and ignition” that is not reliably detected by any fail-safe.

That's proof, not an argument that they could have tried harder to find the system could fail. The bottom line is that its software that puts people's lives at risk. It's reasonable to hold that type of code to a higher standard. There are millions of other cars, trains, and planes out there with similar software but without this type of problem. At some point you should be responsible for the things you create.

More Details

[rabtech]

Couple of details here:

Toyota had no software testing procedures, no peer review, etc. The secondary backup CPU code was provided by a third party in compiled form, Toyota never examined it.

Their coding standards were ad hoc and they failed to follow them. Simple static analysis tools found massive numbers of errors.

They used over ten thousand global variables, with numerous confirmed race conditions, nested locks, etc.

Their watchdog merely checked that the system was running and did not respond to task failures or CPU overload conditions so would not bother to reset the ECU, even if most of the tasks crashed. Since this is the basic function of a watchdog, they may as well not have had one.

They claimed to be using ECC memory but did not, so anything from single bit errors to whole page corruption were undetected and uncorrected.

A bunch of logic was jammed in one spaghetti task that was both responsible for calculating the throttle position, running various failsafes, and recording diagnostic error codes. Any failure of this task was undetected by the watchdog and disabled most of the failsafes. Due to no ECC and the stack issue below, a single bit error would turn off the runnable flag for this task and cause it to stop being scheduled for CPU time. No error codes would be recorded.

They did not do any logging (eg of OS task scheduler state, number of ECU resets, etc), not even in the event of a crash or ECU reset.

The code contained various recursive paths and no effort was made to prevent stack overflows. Worse, the RTOS kernel data structures were located immediately after the 4K stack, so stack overflows could smash these structures, including disabling tasks from running.

They were supposed to be using mirroring of variables to detect memory smashing/corruption (write A and XOR A to separate locations, then compare them on read to make sure they match). They were not doing this for some critical variables for some inexplicable reason, including the throttle position so any memory corruption could write a max throttle value and be undetected.

Instead of using the certified, audited version of the RTOS like most auto makers, they used an unverified version.

Thanks to not bothering to review the OS code, they had no idea the OS data structures were not mirrored. A single bit flip can start or stop a task, even a life-safety critical one.

These are just some of the massive glaring failures at every level of specifying, coding, and testing a safety-critical embedded system.

I am now confident in saying at least some of the unintended acceleration events with Toyota vehicles were caused by software failures due to gross incompetence and negligence on the part of Toyota. They stumbled into writing software, piling hack on top of hack, never bothering to implement any testing, peer review, documentation, specifications, or even the slightest hint that they even considered the software something worth noticing.