Slashdot Mirror
Toyota's Engineering Process and the General Public
Doofus writes "The Washington Post has published in today's paper an article titled 'Why it's so hard for Toyota to find out what's wrong' by Frank Ahrens on the Toyota situation and the difficulties of adequately conveying to Senators and Representatives — most of whom are non-technical — the debugging process. Ahrens interviews Giorgio Rizzoni, an 'expert in failure analysis' at Ohio State, who describes the iterations of testing that NHTSA will likely inflict on the Toyota sample cars they have purchased, and then moves into the realm of software and systems verification: 'He explained that each vehicle contains "layers of computer code that may be added from one model year to next" that control nearly every system, from acceleration to braking to stability. Rizzoni said this software is rigorously tested, but he added: "It is well-known in our community that there is no scientific, firm way of actually completely verifying and validating software."' Ahrens ends the piece with a quote from a 2009 LA Times interview with former UCLA psychology professor Richard Schmidt about how user reports are often unreliable: 'When the driver says they have their foot on the brake, they are just plain wrong. The human motor system is not perfect, and it doesn't always do what it is told.'"
Toyota is currently planning an event to challenge evidence presented by professor David W. Gilbert that called into question Toyota's electronic throttle system.
> Toyota is currently planning an event to challenge evidence ...
Macroscopic events generally don't challenge evidence. They challenge the politics of evidence.
One challenges evidence with small, discrete, verifiable events.
-- IANAL, this isn't legal advice, and definitely isn't legal advice for you. Also, Squee!
Don't get me wrong, it would be incredibly cost, labor and time expensive, and require real computer scientists, but it is certainly possible.
Speaking as a "real" computer scientist, I think you might have underestimated the time requirement. Most problems in automatic verification are either undecidable, or intractable.
An old-timer with old-timey ideas.
Why exactly is there a congressional case going on about this? It becomes even more worrying when you realize that the US government has a controlling interest in most of Toyota's competitors in the USA. In short, why, in a country where states are going bankrupt, privacy is an illusion, healthcare reform has boiled down to if you are pro or anti Obama, rampant spending and tax increases. In short, why do I care about this? File a class action lawsuit and let the courts settle it. Nothing is worse then a bunch of politicians knowing nothing about engineering, with stock in competitor's companies and large problems they haven't solved wasting their time with this crap.
Taxation is legalized theft, no more, no less.
When the driver says they have their foot on the brake, they are just plain wrong. The human motor system is not perfect, and it doesn't always do what it is told.'
This was true with Audi in the 80's, when 60 Minutes did a report where, among other things, they faked a car accelerating out of control (the car was modified extensively.) And yes, a large number of drivers, particularly the elderly, hit the wrong pedal all the time.
However, there are cases where driver reports are plenty accurate. A great example of this would be the problems Volvo V70R and S60R owners have with brake failure while going up hills.
I've experienced it three times in the 6 months or so that I've owned my car. Each time, I was headed up a hill towards a stop sign, put my foot on the brake, and there was nothing there- I had to push so hard I was pulling against the steering wheel for leverage. This is a car with big, high-performance brakes that can stop on a dime.
Volvo claims there's no problem, despite numerous reports on the V70R.com and Swedespeed forums. No other models demonstrate the behavior.
Please help metamoderate.
There's even hardware to do it. dSpace sells some very nice (and very expensive) hardware to do testing. You can setup scripts to test almost any scenario. It'll fake out all the basic sensors and then you can test to see what happens when you hit the brake at 10 mph, 20 mph, 30 mph. You can do burn in tests. Software is very very repeatable. You can often trace right through the Simulink model and find out what is going on.
In the latest versions of CANape you can even view your Simulink Model EXACTLY how you built them and add all of your signal channels to it. If there is a bug or people are experiencing problems, it takes all of an hour at most to figure out what is going on and what is causing it.
And given the short cycle time, you don't have time to rewrite everything. Every company that uses Simulink for models even has verified and validated library blocks. We have a "C to K" block (because one isn't built in). That automatically matches In & Out data types, etc. We have low pass filters that are designed to our companies standards....
And we have engine control models that have been ported from Assembly that have been used for 30 years that 'work'. We're not going to throw that all out the window every development cycle.
Previous comments on how Simulink is used to write code in companies that use it.
SAE Paper on how Caterpillar uses auto coding generation to write their stuff.
I find it odd that the systems in vehicles do not have a default "debugging" which should basically trigger the vehicle to stop.
Why does the vehicle ABS (from what I know from the news) get tripped up on instant breaking? Really? ABS... the thing that is supposed to pump the break to allow for cleaner stops triggers breaking problems and increased acceleration?
I just think bad coding in general here. Regardless of "testing"
Of course Toyota is right. The most likely cause of these "sudden acceleration" problems is humans with their foot on the gas pedal. I've owned plenty of Toyotas, and I wish that my current Toyota was in need of replacing right now, because now is a great time to buy one. Unfortunately, my current Toyota only has 150K miles, meaning that I have a good 5-10 years of life in my vehicle. After that... I'll buy another Toyota.
I don't respond to AC's.
... being in control of braking and acceleration.
If you disagree with me on social issues, then it's pretty clear that you are a narrow-minded bigot.
It is well-known in our community that there is no scientific, firm way of actually completely verifying and validating software.
Looks like Toyota's suffering from a halting problem. ;)
Less than 100 cars out of 8,000,000 have had this problem. That is a 0.001% failure rate.
Of those 0.001% of cars that had the problem, how many times did someone drive them before they failed?
I don't want to say this is user error, but I have seen some users do stupid stuff and not even know they did it.
Im a gamer, not a grammer major. This post is full of spelling and grammer mistakes.
General Motors has been making cars with poor reliability literally since I was a child. Read your library's old copies of Consumer Reports for verification.
Insufficient attention was given to the poor reliability of G.M. cars, in my opinion.
As long as G.M. cars could continue to be sold, making unreliable cars was more profitable. That's similar to making a sloppy computer operating system that is vulnerable to attacks. The sloppiness helps sell new versions.
If neutral won't work- you can also turn off the ignition, but don't turn the key completely off, or you'll engage the steering lock(ie, go to the 'accessory' position.) You will not "lose steering"; at any speed over about 2-3MPH, steering assist becomes less and less necessary, particularly if you don't have very wide tires.)
If you "ride" the brakes, the pad and rotor will heat up and "cook"; consumer, mass-market pads are designed to have good "cold" (ie instant) grab, be easily modulated, quiet, not cause excessive wear on the rotor, and not generate brake dust that is impossible to remove from the wheels. Racing pads are designed for higher temperatures (where among other things, you get much more heat transfer from the rotor to the air blowing past/through it), but they have very lousy "cold" bite. Also, heat up the calipers enough, and you will cause the moisture in the brake fluid to boil (your brake fluid should be changed at a MINIMUM every 2 years, because it is hygroscopic), and that boiling will result in "vapor lock"- no brakes. The brakes MUST be bled after such an incident.
Audi successfully defended itself from several lawsuits and even won a countersuit in a case where a mother crushed her boy against their garage wall (after going through the garage door!). Interviewed by an officer afterwards, she repeatedly said she'd hit the wrong pedal. They sued a few months later claiming the car had "gone out of control". As someone who knows Audis well, particularly the mid-80's 5000 turbo series- the idle stabilization valve (the only way the car computer can increase engine speed) simply cannot allow enough air to bypass the throttle enough to cause the car to lay down burnt rubber, crash through a garage door, and embed itself in a house wall.
The problems with the Volvo "R" models have been reported in a number of other european cars; you'll also see the words "ice mode" thrown around occasionally. Many ABS controllers since 1990 or so have an accelerometer to detect when all the wheels stop simultaneously but there is no corresponding negative acceleration. "Ice mode" is supposedly some sort of variant of this, and there has been great debate as to whether this "mode" is internet folklore, but you'll find many, many posts on all sorts of varying car enthusiast forums.
Please help metamoderate.
The most relevant thing I've read about the problems with Toyota vehicles is this quote from the bottom of page 3 of that PDF linked above:
"... it was determined that [Toyota] Electronic Control Module (ECM) malfunction detection strategies were not sufficient to identify all types of fundamental APP sensor and/or circuit malfunctions. Some types of Electronic Throttle Control (ECT) circuit malfunctions were detectable by the ECM, and some were not. Most importantly, the Toyota detection strategies were unable to identify malfunctions of the APP sensor signal inputs to the ECM. APP sensor signal circuits must be undeniably correct to electrically convey the appropriate driver commands to the ECM."
Next paragraph:
"With the two APP sensor signals shorted together through a varying range of resistances, all four Toyota vehicles tested thus far reacted similarly and were unable to detect the purposely induced abnormality. The types of signal faults introduced into the APP circuit should have triggered the vehicles' ECM to illuminate a warning lamp within seconds."
Bottom of page 4:
"In addition, the shorted APP signal circuits were connected momentarily to the sensor's five-volt supply circuit with the vehicle in drive. In all test vehicles, the ECM did not set a DTC and the engine speed increased rapidly to full throttle. This result shows that unusual or sudden unintended acceleration of the vehicle was possible in the ETC test vehicles."
Erroneus wrote:
Wow. Just wow. Never has a nick been so apt.
This isn't a Toyota thing. It isn't even exclusive to the auto industry. System complexity was where so many cliches like "Fast, complete, cheap: pick any two" come from.
Sure, we can put missile-guidance software protocols into all sorts of software development; If I remember the metric, every line of code costs 10x as much as in general industry.
Another thought: Airbags took 15 years to get acceptance from their 1970's invention -- the industry quickly realized their safety value, but nobody wanted to pony up $800 (1980 estimated per-car cost) or increase the cost of a car to eat that cost.
And don't even get me started on FAA vs. adequate safety. Or Seldane and the FDA.
tl;dr: Toyota *DOES* test extensively. Shit happens.
Oh, dear, dear, dear. Have you evern _looked_ at the details of the TCP protocol, or how and why RAID works? It's only in a non-existent universe with point sources, frictionless bearings, and perfectly spherical fields that such mathematical precision is completely reliable. Even then, the 3-body problem has _not been solved_, nor is the Schrodinger equation easily solved for even the smallest circuits.
So in the real world, "butterfly effects" of small, difficult to predict and model events can cascade into profound changes in quite large-scale systems. Digitization can help, by driving most such effects below the necessary thresholds to turn a bit "on" or "off", but it's not perfect. And mathematical models of mechanical systems are profoundly _not_ perfect: the actual shape of a piece of metal after manufacture, and especially after changes are made after the original design for expense or other manufacturing reasons, can profoundly change the behavior of the real system produced.
Even with software, unless people can follow the code end-to-end, it's prone to surprising errors. Rounding errors, for example, can creep in. Values that are not tested for because one computer scientist read the API one way, and the other read it another way, are rife, and can be be very difficult to avoid.