Toyota Acceleration and Embedded System Bugs

An anonymous reader writes "David Cummings, a programmer who worked on the Mars Pathfinder project, has written an interesting editorial in the L.A. Times encouraging Toyota to drop claims of software infallibility in their recent acceleration problems. He argues that embedded systems developers must program more defensively, and that companies should stop relying on software for safety. Quoting: 'If Toyota has indeed tested its software as thoroughly as it says without finding any bugs, my response is simple: Keep trying. Find new ways to instrument the software, and come up with more creative tests. The odds are that there are still bugs in the code, which may or may not be related to unintended acceleration. Until these bugs are identified, how can you be certain they are not related to sudden acceleration?'"

Comments

Impossible to test

[Darkness404]

Most software is nearly -impossible- to test under flawless conditions. Especially embedded systems with small amounts of CPU power and memory.

Plus, all this hype around these Toyota acceleration problems is just that, hype.

Re:Impossible to test

[Darkness404]

...And if you look at the facts, you can see that all of the symptoms could easily be caused by driver error. Look at this http://www.nytimes.com/2010/03/11/opinion/11schmidt.html?scp=1&sq=driver%20error&st=cse (currently the page doesn't need registration, your results may change in the coming days/hours).

Re:Impossible to test

[Loconut1389]

Indeed. I've done some embedded work myself. I wrote a power supply controller that used DACs to trim the voltage using some analog control ports on the DC to DC converter modules- it also monitored the PowerGood lines on the DC:DC's and linears and was programmed to shut down if one deasserted without a prior command telling it to do so. It had an I2C control network that could request status of bunch of aspects of the board including temperature, voltages, etc. Not wanting to risk blowing out a $10k FPGA with a $4.00 MCU, we had test boards with no FPGA on and some with cheaper FPGAs, and I also had a dev kit with the board on it hooked to a logic analyzer so we could emulate all sorts different scenarios and hopefully protect the FPGAs. Ultimately, a few problems emerged. With a particular combination of testing apparatus and polling rate, the I2C would receive interference and miss or corrupt some data. It was almost impossible to replicate reliably. This in turn exposed an oversight/bug where because of the skipped (as far as the power supply MCU was concerned) bytes, the wrong DAC values were being written, overvolting or undervolting the supplies- but it really only surfaced on the fully populated boards. This lead to a change in the I2C wiring/termination and a move to a keyed and transactional approach that required writing a key value to an address, writing the new data, then optionally reading back the data again, and lastly writing another key to a different address to either commit or roll back. Point is exactly what the parent said, it's very difficult to test some of these things because the problems may be an unusual chain of events or due to very specific circumstance in what's hooked to what and how much power is being drawn in the circuit at the time, etc.

The other portions of the code that performed monitoring and emergency shutdown caught the overvoltages very quickly and shutdown the FPGA in the span of a couple clocks. In the end we only lost one board, and it was due to ESD despite using proper handling techniques and equipment.

Re:Impossible to test

[AigariusDebian]

If a driver dials 911 on his cell phone before even trying to put the car in Neutral, then yeah - it is a driver error.

(The last case on the news - a driver called 911 on his cell phone because his car was accelerating out of control. When prompted by the operator if he had tried putting the car in Neutral, he said no and even refused to do so when ordered to do it by the operator.)

Re:Impossible to test

[Gordonjcp]

Every production car on the road has sufficient braking power to stall the engine in any gear at any throttle setting. Put your foot on the brake, and the car will stop. You may need new discs and pads after that.

Re:Impossible to test

[dbug78]

Here's Mike Allen of Popular Mechanics demonstrating what happens when you shift an automatic Camry into neutral with the gas pedal floored.

http://www.youtube.com/watch?v=NOMYjiCiTYg

It doesn't, in fact, prevent you from shifting to neutral as you can tell by the sound of the engine butting heads with the rev-limiter.

Re:Impossible to test

[konaforever]

Automatic trannies in these cars use the shifter as advice only. If shifted into neutral at speed, the engine would spin out, which is very damaging..

It doesn't seem you know much about cars. First of all, "spin out" isn't an automotive term. And 2nd, what do you mean by "spin out"? Modern cars have a rpm limiter which limits the RPM of the engine to some preset RPM limit. Have you ever driving a manual and hit the RPM limiter? It'll cut power to the engine. Same thing with neutral. An engine will not be damaged if you gas it in neutral, even to the RPM limit.

Re:Impossible to test

[Animaether]

You're one of those types that hit a wikipedia page, see some claimed fact without attribution, slap a [citation needed] on it, and then bugger off, aren't you?

You could just hit a search engine with some key words to see if you can find any corroborating source(s), of course:
http://news.google.com/news?q=toyota%20911%20neutral

Oh hey, look at that.

uring the 911 call, the operator urged Mr. Sikes to shift the car into neutral. He later said he was afraid doing so might cause the car to "flip" or shift into reverse.

- http://online.wsj.com/article/SB10001424052748704734304575120001542947616.html?mod=googlenews_wsj

Re:Impossible to test

[arkenian]

Except that drive-by-wire systems have absolutely no possibility of making the operation of the car safer (more efficient, sure, but that's it) and many many ways in which to make it less safe, so you're wrong about it making more sense to use drive-by-wire. I can't figure out what you're thinking with that statement.

That's flat-out not true. Traction Control and brakes and power synchronizing together (which is a lot of what drive-by-wire DOES) absolutely, in most cases, makes the average driver safer. Personally, I'm inclined to think that as drive-by-wire improves (until eventually, hopefully, the really dangerous part (i.e. the driver) will go away), we will in general progress to fewer accidents per car-mile but, unfortunately, far more catastrophic ones.

Re:Infallible fail.

[shrimppesto]

i'd feel much better with drivers who know they should pop the car into NEUTRAL if it starts accelerating out of control for any reason, rather than trying to stand on the brake pedals while dialing 911 ...

Re:Another interesting statistic

[maxume]

To me it suggests that older drivers are having more difficulty coping with the situation once it arises.

Forbes says that the guy who got himself plastered all over cable last week was 'afraid' to put the vehicle into neutral, or to turn off the engine:

http://www.forbes.com/2010/03/12/toyota-autos-hoax-media-opinions-contributors-michael-fumento.html?boxes=financechannelforbes

(They link the 911 recording:

http://www.thetruthaboutcars.com/the-jim-sikes-911-call-23-minutes-of-unintended-acceleration/

)

So apparently being an idiot is also a likely factor in the failing to cope with the incident before it becomes lethal.

But they key observation is that the higher number of fatalities among older drivers doesn't really point to the source of the problem being driver error (rather, the driver error is in failing to deal with the situation once it arises).

Re:Help me benefit from media hype

[John Hasler]

> And I know how to hit the brakes...

With the engine past the redline there is very little vacuum to operate the power brakes. Without power assist the brakes may not be able to overcome the engine (this is, IMHO, a fundamental design defect).

> ...shift into neutral...

The computer may not let you do that with the car moving and the engine at high rpm. After all, the engine and/or transmission might be damaged (another design defect).

> ...and/or turn off the key...

Some of these vehicles don't have keys: just a radio remote. The emergency shutdown procedure is to hold a button down for three seconds (another design defect).

Re:Help me benefit from media hype

[NewtonsLaw]

> With the engine past the redline there is very little vacuum to operate the power brakes. Without power assist the brakes may not be able to overcome the engine

apparently not true

Re:Boeing versus Airbus

[DerekLyons]

Next week I'll tell you about how the ancient shocking lesson of the British Commet aluminum aircraft wings falling off led to the unanticipated discovery of metal fatigue and probably was the reason Boeing was slow to move to composite materials in commercial aircraft (but not in military aircraft).

This one comment makes me wonder about the veracity of the balance of your account.

1. Metal fatigue was known about long before the Comet took wing in 1949.
2. The fatigue cracks on the Comet's occurred at skin penetrations (windows and hatches).

In hind sight we have heard of many tales of the composite tails of plane falling off as the reason for the loss of control before a crash.

Then there is crunchy bit of FUD, which fails to mention that more than a few of those accidents are also associated with extreme control surface movements (inducing extreme stresses) prior to the failure.

Re:Infallible fail.

[timeOday]

You could be referring specifically to last week's incident, which I found fishy from the start:

Skeptics of Sikes also cite the 911 tape that was released shortly after the incident. During the tape, the dispatcher repeatedly told Sikes to put the car in neutral in order to stop it from accelerating. Sikes did not comply with her instructions or the instructions of the officer on the scene who told him to do the same thing via his public address system as they tore down the highway.... Sikes claimed he thought that would "flip the car."

Beyond the call itself, the Associated Press reports that Sikes's car was equipped with a brake override system, something that should have slowed the car down once he stomped on the brake pedal.