Slashdot Mirror
Toyota's Killer Firmware
New submitter Smerta writes "On Thursday, a jury verdict found Toyota's ECU firmware defective, holding it responsible for a crash in which a passenger was killed and the driver injured. What's significant about this is that it's the first time a jury heard about software defects uncovered by a plaintiff's expert witnesses. A summary of the defects discussed at trial is interesting reading, as well the transcript of court testimony. 'Although Toyota had performed a stack analysis, Barr concluded the automaker had completely botched it. Toyota missed some of the calls made via pointer, missed stack usage by library and assembly functions (about 350 in total), and missed RTOS use during task switching. They also failed to perform run-time stack monitoring.' Anyone wonder what the impact will be on self-driving cars?"
It might change the programming language they decide to use though. Pick a language that's more stable at run-time like Ada (missile programming) etc.
2nd link, 5th paragraph:
Remember when Toyota and DOT concluded the problem was driver error and improperly fitted floor mats?
Any engineering project requires that the engineers have to answer for what they've done. The mantra is, "As an engineer, if you fuckup, someone dies." Every engineer, regardless of discipline, needs to understand this and if they don't, should consider going into Liberal Arts or something equally useless where the worst they can do is fuck up my food or drink order.
some karma... and kinda lukewarm about it.
Vehicle tests confirmed that one particular dead task would result in loss of throttle control, and that the driver might have to fully remove their foot from the brake during an unintended acceleration event before being able to end the unwanted acceleration.
The jury could confirm there was a defect because they were able to reproduce it with a physical car. They could confirm the code quality was poor because it 1) It didn't follow the required code standards: MISRA C, 2) The cyclomatic complexity was too high 3) Toyota didn't track bugs.
I apologize if I'm stating the obvious here...
Most older products were over-built for durability because there were not methodologies for engineering minimum material for the required applications. Cars and other things were built with thicknesses of material that were tested and known to work. To reduce that thickness risked approaching an unknown threshold for failure. Trial-and-error was used where budgets allowed to reduce material, but this was an expensive process and in most cases the manufacturer chose to overbuild.
In more recent years, computer modeling has enabled engineers to load test structural designs so that the product can be built with the minimum amount of material required to satisfy the desired application. This benefits the producer, the consumer, and the scrap yard, while delivering overall efficiency.
$5 / month hosted VPS on linux = awesome!
On airliners they're willing to spend just a little more on extremely reliable and redundant hardware than they are on cars. Makes a difference. It also helps if you code to extremely stringent standards like DO-178B Level A, which costs a fortune. Light aircraft don't use fly-by-wire, why do cars need it?
AFAIK the main argument for fly-by-wire on airliners is that it allows for a reduced stability aerodynamic design, which reduces drag and hence fuel consumption. Considering the amount of fuel an airliner consumes, it's worth spending a king's ransom on fly-by-wire. The payback is definitely there. I know of no similar argument for most of the current generation of electronics in cars, and they're certainly not willing to pay the price.
Seems to me reliability in engine control software _is_ doable. Toyota just didn't do it.
Probably some kind of poor management decision that will ultimately be blamed on bad engineering.
> Again, so far ZERO evidence, proof, or test case has been provided that the software is in any way responsible for this problem.
Just in case that wasn't enough:
Vehicle tests confirmed that one particular dead task would result in loss of throttle control, and that the driver might have to fully remove their foot from the brake during an unintended acceleration event before being able to end the unwanted acceleration. A litany of other faults were found in the code, including buffer overflow, unsafe casting, and race conditions between tasks.
Safety critical systems in automotive applications are fairly rigourous as well. The airbag controller, for example, has a power reserve (a big honkin' cap) so it can trigger the airbags even if the power systems are mangled, dual accellerometers (in case one fails), logging of data, etc.
Brakes are almost always hydraulic with a mechanical backup - malfunctioning ABS cannot defeat the system, etc.
The ECU may not be redundant, but it doesn't matter because if the ECU fails, the engine dies and you try to pull over safely. (in aircraft, you don't want engine failure due to computer failure, so they require dual computers, or computer/magneto).
And fly-by-wire on military jets lets you have better dynamic stability because an unstable aircraft maneuvers faster. Commercial jets are traditional stable designs to begin with. The reason they went fly-by-wire was wire is a LOT lighter than miles of cables, rods, pulleys, hydraulic fluid, etc and has way less error modes (a cable system can fail simply because someone forgot to balance the lengths properly), and makes mechanical assistance much easier to do.
Airbus uses it to avoid having pilot inputs exceed the flight envelope as well.
I've been reading the transcript. It's fantastic. The expert explains clearly and lucidly in terms that (I imagine are) understandable by non-techies.
The transcriber made some funny mistakes... Let me tell you about "parody bits" and "pointer D references" :)
You'd lose that bet.
And likely only once.
http://www.youtube.com/watch?v=xtxd27jlZ_g
No, you'd be experiencing far greater acceleration forces, as if no portion of the car gives way and soaks up kinetic energy, a greater portion of it will be transferred to anything not bolted securely to the frame (eg: you).
The cabin is under no circumstances a crumple zone. Engine and trunk compartments make great crumple zones. The cabin should be a vehicle's Waterloo.
They found numerous bugs, the most damning of which was the fact that recursion (which they shouldn't have had in the first place) was eating up the stack, which wasn't protected at all so that when it overflowed it started trashing other memory. Once this occurred it's anyone's guess what could happen. The stack contains not only local variables but also tells where to return after executing a subroutine. If the stack gets corrupted, then every time a process attempts to return, if that return address (pointer) has been overwritten then you're basically throwing a dart at a board full of assembly spaghetti and starting execution from wherever it lands, which could potentially be code (or even data) that was never intended to be executed under normal conditions (think GTA:SA's Hot Coffee). Usually the process will end up jumping to a memory address that contains data, not code (data execution protection is supposed to prevent this, but it obviously wasn't in the design here), and as soon as it tries to execute an invalid instruction code the CPU will halt, but the process can do bad things before it crashes (like send a signal to the throttle for full power).
Yah I had a jammy throttle in a RX7 I used to drive. Whenever the gas pedal started to get sticky it'd be time to pop the hood and spray it with some WD40.
WD means "water displacer", not lubricant. Should have used a lubricant, not a water displacer. I like silicone products for the engine top, but sometimes I'll just use a general purpose grease.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"