Slashdot Mirror
NHTSA Has No Software Engineers To Analyze Toyota
thecarchik writes "An official from the National Highway Traffic Safety Administration told investigators that the agency doesn't employ any electrical engineers or software engineers, leaving them woefully unable to investigate correctly what caused the most recent Toyota recall. A modern luxury car has something close to 100 million lines of software code in it, running on 70 to 100 microprocessors. And according to consultant Frost & Sullivan, that number will rise to 200 to 300 million lines within a few years. And the software that controls the 'drive-by-wire' accelerators of Toyota and Lexus vehicles is one potential culprit in the tangled collection of issues, allegations, and recalls of many of those vehicles for so-called 'sudden acceleration' problems."
... there is plenty of talent out there for them to hire - even if only on a project by project basis.
Here comes DO-178B for cars.
I wonder what the cost is per line of code?
There are a huge number of yeast infections in this county. Probably because we're downriver from the bread factory.
Surely it would be a serious inefficiency for NHTSA to maintain on staff a large number of specialists to handle this kind of problem? Isn't that exactly what (properly qualified) consultants are for?
Such is the cost of more complicated technology. Although, I will admit, this problem seems awfully widespread for Toyota to have not caught this at some point in their QC/QA process.
I'm reminded of the "recall" speech in Fight Club...
Living With a Nerd
If the statement in the article is true then this country is in even worse shape than I thought. It seems like rarely a handful of months can go by without the realization that yet another Federal department is completely incompetent. How in the hell does the NHTSA even do their job?! They are supposed to ensure that vehicles are safe but they don't even have the staff to do that.
What the hell is wrong with our country?
They don't need Electrial Engineers or Software Engineers. They need Computer Engineers, people who are trained to understand both sides of the hardware/software boundary.
"This mission is too important to allow you to jeopardize it." -- HAL
What exactly would the NHTSA do with a set of engineers? Audit all 100 million lines of code for each and every car they suspect has a safety issue with the computer system? Yeah, that sounds like a worthwhile endeavor. How about they do it the old fashioned way; collect the reports, identify the risk, and sanction the manufacturer to find/fix the problem. Thinking that an NHTSA coder (or a hundred) would have gotten to the bottom of this Toyota issue in any reasonable amount of time is a joke!
sadly, it appears to be true:
http://spectrum.ieee.org/green-tech/advanced-cars/this-car-runs-on-code
If the NHTSA didn't exist Toyota would have had to spend money to fix the problem instead of paying ex-regulators to quash multiple investigations.
Toyota (TM) hired ex-government regulators to kill at least four investigations into problems with its cars in the U.S. That's the conclusion of an investigation by Bloomberg. The news service reports that, "Christopher Tinto, vice president of regulatory affairs in Toyota's Washington office, and Christopher Santucci, who works for Tinto, helped persuade the National Highway Traffic Safety Administration to end probes including those of 2002-2003 Toyota Camrys and Solaras, court documents show. Both men joined Toyota directly from NHTSA, Tinto in 1994 and Santucci in 2003. "
The same goes for Wall Street. Most of the financial regulators are former high level executives from Goldman Sachs or strong ties to them and other financial institutions.
I don't understand why we need so many useless regulators who are usually wolves being put in charge of the hen house when the courts could easily handle this. It's going to end up being prosecuted in a court of law anyway and not solved by some magic regulation hand-waving.
If you have something that you dont want anyone to know, maybe you shouldnt be doing it in the first place -Eric Schmidt
http://abcnews.go.com/Blotter/RunawayToyotas/toyota-acceleration-problems-new-evidence-imprisoned-minnesota-toyota-camry-owner/story?id=9903455
This guy apparently killed a few people and got put in jail for it. Now it looks like he was telling the truth when he said the car wouldn't stop.
Mod me down, my New Earth Global Warmingist friends!
Anything street legal without a needing a special waiver for emissions.
Go to a car dealer. Look. Every car sold since 1996 (At least in the US, and I assume the rest of the world) today has at least an ECM (Engine Control Module) which is just a fancy name for a computer controlling the engine. That's what the government mandated OBD-2 program was (OBD == On Board Diagnostics). The number of cars that are completely computer controlled (drive by wire) is far lower, but higher than you'd think.
I had an '05 Chevy Cobalt that had "computer assisted" electromechanical power steering. Basically, what I found out from the dealer after the computer controlling it failed (and I lost all power steering) is that the computer (BCM, Body Control Module) takes inputs from the ABS system, Traction control (if equipped), speedometer, accelerometers and about a dozen other sensors and computes the way it thinks you want to be steering. Then it provides an "intelligent" boost in that direction. I must say, it worked really well in the snow and when fishtailing (it made if VERY difficult to over-correct and put it into a spin). But when it failed, I'd be in the middle of a curve on the highway when all power steering went out... Luckily they were smart enough to put a kill switch in to prevent it from coming back on while the car was moving (I could just imagine struggling through a corner when all of a sudden it came back)... It turns out that it was a software issue in the first place (they updated the software, and it never happened again). I got rid of the car a few years later for other, more significant reasons...
The benefits of computer control are good, but there needs to be intelligent fail-safes put in place to prevent disaster when something does go wrong (not if, when)...
If a man isn't willing to take some risk for his opinions, either his opinions are no good or he's no good
I strongly suspect the "100 million lines of code" is BS. Most of the "ECUs" are small microcontrollers that would be lucky to hold 5,000 lines of code, let alone millions. Either the professor is inflating the code size estimate to make himself seem important, or the systems are designed by complete idiots.
Most of that code is auto generated. Except for some low level stuff, nothing is written by hand in assembly or C. It's all auto coded from some sort of control toolbox. Most likely Matlab/Simulink.
Sure enough this is one of the first hits on Google.
Writing that many lines of code would be damn near impossible in the relatively short development cycle.
Even a simple PID controller could take up a few dozen lines of code even though on screen it's simply represented by 3-4 blocks.
And they said in a modern luxury car.
So that's all the code in the following computers:
Engine (controls throttle and such)
Transmission
Collision avoidance (ABS, traction control, etc. TPMS is usually here, too, because it's sometimes part of the ABS system to save costs)
Safety (airbags, seatbelt pretensioners, etc.)
Central convenience (security system, power locks, power windows, cabin illumination, in some cars even the exterior lighting goes through central convenience)
HVAC
Instrumentation (yep, there's a computer dedicated to that - and some security functions are sometimes in there)
Entertainment (navigation, stereo, DVD, etc., etc.)
And all these systems are interconnected.
You get in your car (central convenience deactivates security upon receiving the signal, and when you open the door, it illuminates the cabin, alerts the engine computer that a start is imminent, possibly starting fuel pumps, on diesel cars turning on the glow plugs, etc., etc., and notifies the instrument cluster that the door is ajar.)
You insert your key into the ignition (yes, I know about push-button start,) and start the engine (engine computer starts up, after which the instrument cluster polls the RFID chip on the key. If it can't get a read, it immediately requests that the engine computer shut down.)
You decide that you want a little heat before you set off, so you use your steering wheel controls (which go through instrumentation) to set HVAC settings, and then you figure some music won't hurt (entertainment.) Then, you remember that you don't know where you're going, so you punch the address into the navigation system, and it feeds directions back to the instrument cluster.
Now, you put the car into gear. The transmission computer notifies the other computers about this, and the engine computer adjusts the idle fueling to compensate. The instrument computer reflects the gear change. The central convenience module turns on the daytime running lights. The entertainment system might prevent you from using the touchscreen interface. The safety computer may become more persistent about reminding you that you didn't put on your seat belt, and will notify the instrument cluster of this, to annoy you more.
After you put your seatbelt on, you let off the brake and pull out of your parking space. Obviously, the engine computer and transmission computer are working together here, the instrument cluster is constantly updating the status of those (and the entertainment computer, which is noting the changes in vehicle position.) After you hit 10 MPH, the engine or transmission computer sends a request to the central convenience module to lock the doors.
Now, you're going down the freeway, and right in front of you, a semi truck loses control, and flips onto its side. You jam on the brakes, which kills engine power immediately (engine computer, and the transmission computer is affected as well, and this all gets fed back to the instrument computer.) Collision avoidance computer activates ABS and (as you're attempting to swerve out of the way) stability control, and notifies the central convenience computer that you're undergoing a panic stop, and to activate the hazards.
Unfortunately, you don't have enough time and room to stop, and you hit the semi. The safety computer notices this, and fires the seatbelt pretensioners and the appropriate airbags. Once that's done, there's some less immediate concerns. It would be a bad idea to leave the engine running, so the safety computer requests an engine shutdown. The transmission computer may be requested to shift to neutral, to make moving the wreck easier. The entertainment system will be told to stop playing music, and if it's got a system like OnStar (which used to be yet another TWO separate computers off of the entertainment system,) an emergency call initiated. Instrumentation is of course updating the status of all of this. HVAC may be set to off. The collision avoidance computer will still be trying to keep t
Here comes DO-178B for cars.
The vehicle drivetrain network is very often, if not always, separate from the "entertainment" network; Audi, for example, runs two separate CAN busses for them. The original story hypes things a bit; there may be 70-100 microCONTROLLERS, but half or more of them are "body" (ie windows, sunroof, etc) or "entertainment"(audio, navigation) related and thus don't really need to be reviewed.
The vast majority of them do very, very simple things, mostly sending CAN bus messages or responding to CAN bus commands. Ie, you move the wiper stalk. The microcontroller for the steering wheel controls says "the stalk moved" either to the wiper motor interface or a 'body control' computer, which then sends a command to the wipers.
The code review for most of the modules, as a result, is extremely simple- they're just (mostly digital) I/O boxes. Some of them are things like fuel pump modules, which at most have some diagnostic capabilities (like current draw from the pump, pressure sensor, etc.)
The code review will not be very problematic for engine computers, because (gasp!) they're not made by car manufacturers. Bosch, Magnetti Marelli, Hitachi, and a couple of other companies are the primary producers. And guess what? The code is largely the same car-to-car. Parameters are changed- code doesn't, so much. And car companies share "platforms", which further simplifies things.
It's not nearly as scary as it sounds.
Please help metamoderate.
Shift into neutral. I haven't seen this anywhere as part of the many Toyota-related discussions around the world, so figured I'd mention it.
Palaces, barricades, threats, meet promises
1. A car designed for manual steering is quite different than one designed for power steering.
2. There is a wide range of speed and turn radius conditions between straight freeway and parking lot.
Most of the financial regulators are former high level executives from Goldman Sachs...
Some are but most are demonstrably not. Many are financial industry insiders but that's by necessity. Do you really want an financial regulator who has no knowledge of the industry he/she is regulating? The only place to get people with the appropriate financial experience is from the finance industry.
I don't understand why we need so many useless regulators who are usually wolves being put in charge of the hen house when the courts could easily handle this.
While I admire your faith in the court system, in truth the courts are woefully ill-prepared to deal with the sorts of issues the SEC and other regulating bodies deal with. The court system is sloooooowww, expensive and can only effectively deal with misconduct after it has occurred. The courts are a poor monitoring system. The court system also is not heavily staffed with financial experts who understand the issues involved. Trust me, you REALLY don't want financially illiterate judges deciding financial regulations.
The reason the industry insiders often end up as regulators is precisely because they are the only ones who really understand what is going on. Finance is really, really complicated. Yes it's not perfect but that's why the regulators are accountable to other bodies including the President and Congress. If anything the problem with the regulators isn't (usually) that they do poor quality work but rather that they aren't given enough resources to really do a great job. The SEC for instance is badly understaffed given it's mandate. If you really want to keep a better watch on the finance industry, lobby congress to increase funding to the SEC and other watchdog agencies.
It's going to end up being prosecuted in a court of law anyway and not solved by some magic regulation hand-waving.
Spoken like someone who has no experience whatsoever in the financial industry. I won't argue that all regulations are good or well enforced but relying on the court system alone to solve the issues that regulators deal with daily would be insanity. If you really want to screw up the financial system, get rid of the regulators. Our current financial mess is due in significant part to a lack of regulation.
It would be a pretty crappy car if it engaged the seat belt PREtensioners POST-impact.
Pretensioners are fired after the initial contact, whilst the very front of the vehicle is still crumpling away. How the hell do you think the computer knows that it has hit something otherwise? Radar? Not on your $10K cheapo. Magic? No, a little ball + spring combo live underneath your front bumper and the last thing they tell the vehicle before they are crushed in an accident is "something big is heading your way".
I'd also rather it didn't "kill engine power" every time I hit the brakes.
We're not just talking about 'every time', we're talking about the two-feet-on-the-brake-pedal-jesus-christ-I-want-to-stop-NOW kind of braking that will activate ABS. Once ABS (and it's cousin, stability control) are running the show, engine power can (and will) be modulated as they see fit in attempt to keep the vehicle going where you want it to go. If you think you can simultaneously control brake force and engine power separately to each wheel whilst in an emergency to do the same, than you go right ahead. I'll take the bus.
although "traction control" systems might retard timing if severe wheel slip is detected.
Traction control is a lot smarter than you seem to think now, and retarding timing went out of fashion about 15 years ago. Now if the traction control system wants less power it simply requests the engine computer to reduce power output by X percent and the engine computer will choose between:
- Simply closing the throttle body, if it has control of it.
- Killing fuel injection on a few cylinders to drop power.
- Dropping boost if it's a turbo'd vehicle.
- Cutting (or yes, retarding) ignition. Bit of a last resort due to unburnt fuel getting out the other side of the engine.
And what $20,000 compact automatically turns on hazard blinkers, mutes the stereo, and opens windows?
My Peugoet 307 turned on the hazards and muted the music if you hit the brakes hard enough to activate its electronic brake force assist system. I did it a couple of times in the two years I had the car, but never got into a collision to find out about the windows.
You are in a twisty maze of processor lines, all alike.
There is a lot of hype here.
The government doesn't have to do anything complicated. It just has to have the ability to strike fear into the hearts of the business community it's supposed to regulate.
This requires a few things: an independent media, which we don't have; a civically informed populace that takes it's democratic duties seriously, which we don't have; and a culture that values human dignity over profits, which we don't have.
In cultures that do have all of these things, government regulation works very well and fosters progress, since you don't have to constantly worry about getting screwed over, you don't have to wonder if you'll have access to medical care, or a good public school, or a good safety net to get you back on your feet if your fall ill, get in an accident, or whatever.
Clear and concise regulation with real penalties for breaking those regulations fosters competitive markets. Diminishing the government to the point where it can be bought and sold by businesses usually leads to fascism. The markets destroy themselves with greed, destabilize the economy (and eventually the whole society), and further concentrate wealth and power until you have a virtual oligarchy sprinkled with political theater.
Why not simply require that any software in an automobile be OSS (not FOSS). In fact that requirement should seem to be an extension of mechanic laws that required car makers to provide parts and knowledge to service vehicles outside dealerships. All software in such a critical item should be OSS so it can be reviewed for errors and be reprogrammed by mechanics who wish to offer such services.
And safety, not peformance.
Instead of testng code, evaluating the design process, pretending the NHTSA can even begin to become expert in software design, how about applying the old standards to the new systems?
For instance, braking safety. I was listening to and reading the testimony from Rhonda Smith, where she even describes shifting her Lexus into neutral. Neutral?
A simple test, and I'm not an engineer, but shouldn't a car come to a stop with 'maximum' brake effort, despite the acclerator position? This is solvable in software - if the brakes are going into lock, and ABS is engaged, engine power and/or transmission state have to be compelled to answer the driver's command to stop. Traction control is already being used in many cars; NHTSA should be able to make a test capable of verifying that even multiple malfunctions are overcome.
Crap, my wife's 1995 Saab 900SE has a mode where the ECU shuts down the fuel pump if the engine stops running, on the assumption that something is terribly wrong, and spewing gas to a stopped engine is pointless if not dangerous. How do I know this? Her car developed a habit of stalling at stops. The real cause was a defective vapor recovery canister, causing loss of vacuum and low RPMs, and the ECU saw that as a stopped engine and made sure it stopped.
Certainly there are other states that can be tested for performance and safety, not some quality of performance standard. Most cars have 'safe' or 'cripple' modes to protect the drivetrain if something seems wrong, like the transmission in a gear that should not permit the indicated speed. My '95 Explorer does that, and it's only an OBD-I system. Acclerator position, wheel speed, and transmission mode should all correlate, and if something is wrong the system needs to cripple - slow down, set a max speed, etc.
Aircraft flight control systems are held out as an example of safety and reliability. Most of these, if not all, have to at least ensure the aircraft doesn't exceed the flight envelope and exceed safety limits. This is the sort standard and evaluation the NHTSA needs to focus on.
Maybe NHTSA needs to borrow a few investigators from the FAA and the military? They should be looking to Boeing, McDonnell, Electric Boat, General Dynamics for expertise in verifying safety in vehicles. Maybe even some NASA people. At least NASA seems to have turned the Shuttle program around a little too late. They certainly have a cautionary tale to tell, and a jaundiced eye towards the assurances of the 'experts' and trusting management.
Which would go a long way to reinstating a somewhat adversarial relationship between the regulators and the industry. There should be some tension there. Hiring your industry's former employees is not the way to go.
We can do so much better. We just need to solve the real problems.
deleting the extra space after periods so i can stay relevant, yeah.
Here's some recent data about the resources available to the DoT, the parent agency of the NHTSA: When the recession started, the Transportation Department had only one person earning a salary of $170,000 or more. Eighteen months later, 1,690 employees had salaries above $170,000. Plus the juicy benefits and pension plan. I'll bet all those managers and supervisors raking in the big bucks would agree that their agencies are "resource starved" and that if they only had more money and more power, they could hire two or three software engineers (for the cost of one manager).
Q: What does the "B." in Benoit B. Mandelbrot stand for? A: Benoit B. Mandelbrot
It's not a race condition, is it?
I can't believe I'm the first one on this thread to make that joke. I'm not even a programmer.
You should all be ashamed of yourselves.
Finally had enough. Come see us over at https://soylentnews.org/