Toyota Acceleration and Embedded System Bugs

An anonymous reader writes "David Cummings, a programmer who worked on the Mars Pathfinder project, has written an interesting editorial in the L.A. Times encouraging Toyota to drop claims of software infallibility in their recent acceleration problems. He argues that embedded systems developers must program more defensively, and that companies should stop relying on software for safety. Quoting: 'If Toyota has indeed tested its software as thoroughly as it says without finding any bugs, my response is simple: Keep trying. Find new ways to instrument the software, and come up with more creative tests. The odds are that there are still bugs in the code, which may or may not be related to unintended acceleration. Until these bugs are identified, how can you be certain they are not related to sudden acceleration?'"

Comments

Impossible to test

[Darkness404]

Most software is nearly -impossible- to test under flawless conditions. Especially embedded systems with small amounts of CPU power and memory.

Plus, all this hype around these Toyota acceleration problems is just that, hype.

Re:Impossible to test

[Darkness404]

...And if you look at the facts, you can see that all of the symptoms could easily be caused by driver error. Look at this http://www.nytimes.com/2010/03/11/opinion/11schmidt.html?scp=1&sq=driver%20error&st=cse (currently the page doesn't need registration, your results may change in the coming days/hours).

Re:Impossible to test

[Loconut1389]

Indeed. I've done some embedded work myself. I wrote a power supply controller that used DACs to trim the voltage using some analog control ports on the DC to DC converter modules- it also monitored the PowerGood lines on the DC:DC's and linears and was programmed to shut down if one deasserted without a prior command telling it to do so. It had an I2C control network that could request status of bunch of aspects of the board including temperature, voltages, etc. Not wanting to risk blowing out a $10k FPGA with a $4.00 MCU, we had test boards with no FPGA on and some with cheaper FPGAs, and I also had a dev kit with the board on it hooked to a logic analyzer so we could emulate all sorts different scenarios and hopefully protect the FPGAs. Ultimately, a few problems emerged. With a particular combination of testing apparatus and polling rate, the I2C would receive interference and miss or corrupt some data. It was almost impossible to replicate reliably. This in turn exposed an oversight/bug where because of the skipped (as far as the power supply MCU was concerned) bytes, the wrong DAC values were being written, overvolting or undervolting the supplies- but it really only surfaced on the fully populated boards. This lead to a change in the I2C wiring/termination and a move to a keyed and transactional approach that required writing a key value to an address, writing the new data, then optionally reading back the data again, and lastly writing another key to a different address to either commit or roll back. Point is exactly what the parent said, it's very difficult to test some of these things because the problems may be an unusual chain of events or due to very specific circumstance in what's hooked to what and how much power is being drawn in the circuit at the time, etc.

The other portions of the code that performed monitoring and emergency shutdown caught the overvoltages very quickly and shutdown the FPGA in the span of a couple clocks. In the end we only lost one board, and it was due to ESD despite using proper handling techniques and equipment.

Re:Impossible to test

[AigariusDebian]

If a driver dials 911 on his cell phone before even trying to put the car in Neutral, then yeah - it is a driver error.

(The last case on the news - a driver called 911 on his cell phone because his car was accelerating out of control. When prompted by the operator if he had tried putting the car in Neutral, he said no and even refused to do so when ordered to do it by the operator.)

Re:Impossible to test

[Gordonjcp]

Every production car on the road has sufficient braking power to stall the engine in any gear at any throttle setting. Put your foot on the brake, and the car will stop. You may need new discs and pads after that.

Re:Impossible to test

[konaforever]

Automatic trannies in these cars use the shifter as advice only. If shifted into neutral at speed, the engine would spin out, which is very damaging..

It doesn't seem you know much about cars. First of all, "spin out" isn't an automotive term. And 2nd, what do you mean by "spin out"? Modern cars have a rpm limiter which limits the RPM of the engine to some preset RPM limit. Have you ever driving a manual and hit the RPM limiter? It'll cut power to the engine. Same thing with neutral. An engine will not be damaged if you gas it in neutral, even to the RPM limit.

Re:Infallible fail.

[shrimppesto]

i'd feel much better with drivers who know they should pop the car into NEUTRAL if it starts accelerating out of control for any reason, rather than trying to stand on the brake pedals while dialing 911 ...

Re:Another interesting statistic

[maxume]

To me it suggests that older drivers are having more difficulty coping with the situation once it arises.

Forbes says that the guy who got himself plastered all over cable last week was 'afraid' to put the vehicle into neutral, or to turn off the engine:

http://www.forbes.com/2010/03/12/toyota-autos-hoax-media-opinions-contributors-michael-fumento.html?boxes=financechannelforbes

(They link the 911 recording:

http://www.thetruthaboutcars.com/the-jim-sikes-911-call-23-minutes-of-unintended-acceleration/

)

So apparently being an idiot is also a likely factor in the failing to cope with the incident before it becomes lethal.

But they key observation is that the higher number of fatalities among older drivers doesn't really point to the source of the problem being driver error (rather, the driver error is in failing to deal with the situation once it arises).

Re:Boeing versus Airbus

[DerekLyons]

Next week I'll tell you about how the ancient shocking lesson of the British Commet aluminum aircraft wings falling off led to the unanticipated discovery of metal fatigue and probably was the reason Boeing was slow to move to composite materials in commercial aircraft (but not in military aircraft).

This one comment makes me wonder about the veracity of the balance of your account.

1. Metal fatigue was known about long before the Comet took wing in 1949.
2. The fatigue cracks on the Comet's occurred at skin penetrations (windows and hatches).

In hind sight we have heard of many tales of the composite tails of plane falling off as the reason for the loss of control before a crash.

Then there is crunchy bit of FUD, which fails to mention that more than a few of those accidents are also associated with extreme control surface movements (inducing extreme stresses) prior to the failure.