Slashdot Mirror
If The Problem Persists, Reboot The Car
prostoalex writes "Ever-increasing presence of high-tech devices in modern cars is a double-edged sword, the New York Times discovers. Software from different suppliers brings up to some peculiar bugs, such as a heater turning itself on during a hot summer day. In December last year ABI Research estimated that roughly 30% of all warranty issues with new cars were microprocessor- and software-related. The NYT article also quotes an interesting prediction from IBM, saying that by 2010 almost all cars will have the same mechanical systems (hardware), and the differences will be primarily on software level." (That prediction seems as accurate as the IBM prediction that there was a worldwide market for 10 or so computers.)
Actually, on my last flight, we were delayed for a couple hours on the tarmac becasue they had to "replace a part". When they put the new part in, they had to turn off and back on the plane, and they waited about 2 minutes between turning off and back on.
It was eerie sitting there for those two minutes thinking, "They're cold booting the plane that is about to take me into the air."
Pulp Audio Weekly - Geek News and Reviews
Personally, I like the way old cars worked. There is no need for a computer in a car. Old cars are easier to fix. They also cost less to fix. They are also immune to electromagnetic pulse should a nuclear bomb go off. Granted, old cars do not have all the fancy navigational equipment and automated devices.
Scary. I just read that joke today. And it was about a Microsoft programmer. It was a link from a site which was linked from someone's post on the Bill Gates interview posted today on Slashdot. (If you can understand what I just said). Scary. The first thing I thought of when I saw "Reboot the car" was that joke.
WASTE - The Secure P2P
As technologies moves away from OEM's and to the Tier Ones, more and more components become the same between automakers. Look at MAF's and Fuel injectors, and O2 sensors, and (fill in blank). They are likely to differ by 1 digit in a part number between different makes of cars (if at all). As sensors and actuators become standardized, the only difference between automakers will be what they do with them (software!!!).
A few months ago, in France, a man driving his car experienced a bug : the automatic speed limitator hanged, and he was stuck on the highway at 200 km/h (125 mph). He called the police while he tried no to collide with trucks, passing them on the right. The police had to open the gates at the next toll to let him pass. After 150 km (90 miles), the car finally stopped. Renault, the constructor, said they detected no problem and sued him (maybe he just lied). Anyway, when you have to get your car repaired, it is longer and more expensive than a few years ago, because of electronic parts being repaired only by the constructor.
which was supposedly "repaired" by the dealership. 2 days after I picked up my jeep, the check engine light came back on.
Rather than take it in for a $90 diagnostic fee, I found a trick: put in key, turn "on", "off", "on", "off", and finally, "on", and the diagnostic code(s) display in the digital odometer.
Take that code to google, and I found that the jeep had the same problem which was supposedly "repaired".
I took it back in, and mentioned the codes it was showing, and put up a fight with the service manager saying, "No, you will not charge me for the 'diagnostic' check, I already know what it is. It's code ABC and XYZ. Now, fix."
I concur with another reply to this comment. I find it interesting we've come to a level of quality in our technology that consumers actually expect and assume things will need to be rebooted. As for the OP of this reply, if one would be expecting and not minding a reboot as normal activity, how would one at the same time expect reliability such as "a lot of redundancy" to co-exist? If something is ill-behaved enough to likely require reboots, it IMO is equally likely to not have reliable redundancy.
In the IT industry, I've encountered people who found something to work reliably 999 times out of 1000 to be good reliability (I used to work at microsoft -- many of them considered that good). But when you start talking about my car, I really think reliability needs to be essentially perfect. God help us if we start seeing this attitude in jet and airplane technology. (If we had .999 reliability in takeoffs and landings, guess how many crashes (literally) there would be every day!)
I worked at a dealership back in 1998, right out of High School. I was amazed, on a daily basis, how far computers in cars had come since my 1979 Volvo had been made.
In a lot of cases, we could tell the telemetry on the car in the last week. Also cars adjusting themselves to the driver. (Pretty annoying when you loan your car to your lead foot mate of teenager).
Cars have, without a doubt, come just as far again since then.
I also have to agree that the mechanics of cars won't be seeing too may improvements in the next decade or longer. The real advances will come in fine tuned computer controlling of all the support systems. Fual, air, spark, all adjusted based on temp, humidity, how hard the driver is pushing the car, and many other factors.
I don't see any reason why a straight gas car can't do over 100 MPG, given the right fuel.
Pretty Pictures!
I was on a 7 pm flight this week. We say on the runway for over 2 hours while a computer on the plane was rebooting repeatedly. Finally they rebooting the entire plane, shutting everything off and then bringing the systems back up. That didn't work either. Finally they gave up, said that this had happened early in the day (!) and put us on another plane going to a different city. To make a long story short, I got in at 2 am instead of 8 pm and I could have driven there faster.
Lasers Controlled Games!
That prediction seems as accurate as the IBM prediction that there was a worldwide market for 10 or so computers.
Why does that seem far-fetched? Cars today are mechanically similar to the Model T Ford.The only changes that have been added over the years have been for the convenience of the driver, like power windows and auto transmissions.
By 2010, there will be more hybrid cars and (as predicted) more computerized cars. Other than that, I don't think there will be any differences. What does timothy think will be different?
Give a man fire, and you warm him for the night. Set a man on fire, and you warm him for the rest of his life.
As an ECU engineer (Chrysler products right now), I tell you that the chances of this happening can be expressed as 1/infinity (0). Well, it is possible that this could happen on older cars, but not newer systems.
A brief overview of the car's computer architecture is needed to understand why. Our cars today are filled with electronic devices that are connected in a network. Think of the car as a computer lab with computers connected via a bunch of different hardware: gigabit ethernet for some, 10/100 for others, and serial connections for some others. Each of these computers has a single purpose and all the computers have to talk to each other to get the information they need to complete their tasks.
On the autos, we call these networks CAN (B/C mostly) (similiar to ethernet, but at 1Mbit max theoretical speed and 500kbps max practical speed, and LIN (along with some others, but I'm not familiar with them) (a sort of serial protocol). We then separate all of the computers on the network, called ECUs, into different levels depending on how critical they are, and how much information they need to relay.
For example, the engine controller is very important to your car, so it goes on high-speed CAN (CAN C) and does not interact with low-speed CAN except via a gateway. As you may have guessed, ECUs on high-speed CAN are audited and tested much more than their low-speed counterparts (ie. you'd be angry if your radio failed, but you'd be fucked if your brakes suddenly stopped working; suppliers test accordingly).
Radios usually include a clock functionality and is used to synchronize various ECUs. Because the high-speed ECUs require much more precise timing, the radio was put on the high-speed CAN bus. So, in older cars, a seriously malfunctioning radio, could potentially saturate the high-speed bus.
Maybe. Because of the way CAN was designed, priority is given to the holder of the lowest message id in the event of messages being sent out on the bus at the same time. So, the important systems like your engine controller and your ABS system would likely have priority over your radio. Unfortunately, something like resource exaustion could possibly take place with an ECU gone haywire and performing a D.O.S., but it isn't very likely.
Recently, this design decision was rethought and we think we have fixed it.
Since it's diesel, there's no ignition system. It did have an alternator, but really it was just there to charge the battery so the starter motor could run, but it came with a handcrank so that if need be, you could handstart it. I did that once. Painful, but possible.
No engine computer. No electricity needed whatsoever. Everything was mechanical.
The timing was done by a cogged system, so there was no timing belt to break. In fact, the only belt in the entire engine was for the alternator. The water pump was also cogged.
The manual had a succinct sentence in the debugging section: "If the engine is getting clean fuel and air, it must run". This was actually a bit pessimistic; one fellow I knew ran out of fuel a few miles away from home once in a flat calm and ended up pouring a mixture of turpentine and paint thinner into his fuel tank. The diesel fired up and run just fine, albeit a bit smokier than usual.
A lot of things are easier to design when they are digital. That way they can get sensors or modules (like a whole dash instrument panel) manufactured to work with a set of standards (protocol). Plus you can use bus sharing. Analog stuff is also prone to calibration (both 0 offset and gain), and are greatly affected by bad connections and such.
... You get better gas mileage out of it, better pollution control, etc. Plus the computer spits out some error messages that help narrow down the problem.
The point is no to computerize the temperature controls, but the engine, transmission,
Cheaper, better... Every way you look at it it's better.
The EVAP system in your car is to prevent release of gasoline fumes from the fuel venting system. A faulty gas cap, or a gas cap not turned tightly onto the tank can indeed cause it to fail diagnostics (because several of the diagnostics involve very slightly pressurizing the fuel tank and then checking for leaks). Also, there can be problems with the actual charcoal cylinder or solenoids that open and close the vent lines. It is not expensive to actually diagnose those vent lines/solenoids. Anyways, the EVAP system is usually close to the fuel tanks and not under the hood.
EVAP is just an emissions issue; it won't actually cause any problems driving around in almost all cases.
I have heard that the reliability of some EVAP systems have been improved by firmware changes that loosen up the diagnostic criteria a bit, because the original diagnostic criteria is a little strict and causes things to fail early.
Once I was on a fancy plane that had LCD Screens in the back of all the seats.
Someone accidently kicked loose the external power feed while the planes APU was the only internal source of power... Lights out (ill tell you, have fun getting out of a burning plane on those emergency lights - make you panic more), you could hear the alarms in the cockpit from the back fo the plane. I could actually people yelling outside, through the skin of the plane.
When they got the power reconnected, i got to watch the computer system reboot on the screens. Guess what was running? Linux.
The problem the NYT is describing here is not high-tech devices. The problem is crappy firmware.
This cannot be over emphasized. Firmware quality in general is declining.
I work for a company, who shall remain nameless that produces embedded devices. Looking at all the different devices, such as the Dish network 522 DVR, automotive electronics, and my company's products, firmware quality in general is on the decline.
There are two basic reasons for this. The general buzzword driven nature of IT, and certifications has marketed an idea to businesses that the geeky elite embedded software developers with their top-notch traditional CS background (whether formal, or self taught... but a lot of management surprisingly doesn't understand the fundamental difference between the kid that was hacking 6502 assembly code and the nerd doing case mods) are no longer necessary. Since we have things like "Embedded Java" and Windows CE with Visual Basic there is a new opinion that ANY Java developer or ANY Visual Basic developer can be a productive embedded software engineer. 15-20 years ago, the typical software engineering lead working for Xerox, or Ford, or Boeing was usually a top notch CS person with an indepth knowledge of hardware and software architecture, data structures, algorithms, and knew something about reliable software design. Now, if personnel sees that your buzzwords: Java (oh, so he can use Embedded Java); C++ (ah, Embedded C++), and honestly what resume does not have C++, from interviews it is clear to me that only a fraction of those that put C++ on their resume can get past "hello, world" with stdio; match some of the mainstream buzzwords, and you have a few years of experience, you can easily obtain a lot of jobs with firms that are writing software for applications that you'd expect a much higher standard of quality then your typical crap.
Now there is this idea that an American or Indian or Chinese firm that advertises Web and Embedded Development is fully capable of turning around high quality bullet proof software.
That is the second reason, outsourcing (not necessarily internationally). Because many American firms hire people with what is essentially an IT tradeschool (my company hired a guy with a Chemical engineering degree, that had attended a few seminars on using Visual Studio... and no.. he is not a geek), which they will use as maintenance personnel, and then they find some firm to outsource the engineering to.. There is no gatekeeper or certification for engineering firms (the PE is a joke (unless you're a Civil Engineer), you don't need to call yourself an engineering firm to still get plenty of business), and the firms are increasingly picked by middle management with very little technical background to make an informed decision on the technical capabilities of the firm (I personally know this is exactly why Dish Network's firmware invariably sucks dick).
But this even happens with desktop PCs, admittedly without the huge risk to human life. No matter what the dialog box, people will click 'OK' or 'Yes'. People don't realise systems give feedback for a reason, and unless you know for a fact the system has got it wrong then you act upon it.
In an ideal world, I would be expecting the plane's computer to be reporting things like this to Air Traffic Control so they can start considering emergency paths even before pilots radio in a problem.
How many people can read hex if only you and dead people can read hex?
I rebooted my car while going down the highway, the CD player stopped working and kept saying Error (2002 Ford Explorer, crappy stock head unit) after we tried to load a CD that didn't go in fully (the motor went but the CD didn't go in.) We were on a road trip following someone so instead of stoping we tried to fix it other ways. We turned the radio on and off, tried switching modes, then eventually shifted the car into neutral, turned it off, and turned it back on. That did it.
regardless of how outrageously stupid customers are, a good helpdesk can learn from them. There is a good reason to know traffic(though even there some automation is possible), but why should a person know if the engine is in the front or the back? Just to be nice, imagine the person is from a country far far away, just got off the plane, and wants to rent a car.
ex 1. after you identified yourself, why doesn't your car start automatically? It could offer you to start.
ex 2. If the gas tank is running empty, the car becomes more insisting. It tells you it needs food. Then it tells you it needs food urgently and the nearest place to get it.
ex 3. ignoring cruise control, what decides the pace of deceleration when you let go of the accelerator pedal? It's not chosen, it's just decided by the construction of the engine. The car could start breaking if you let go of the pedal. If you press the pedal a little bit, you get the previous 'breaking on the engine'. The breaking pedal is for extra breaking.
ex 4. Car interface became more complicated than 30 years ago. Implement an 'automated' option so controls can move out of the way, with manual control as an override. This is what is being done with headlights(sensing light/dark ) and wipers(sensing rain).
Each of these ideas can be dismissed. But hey, imagine getting just one real proposal out of 100 customer questions.
Here is an example of what to expect in a computer-controlled car. I have s 2004 Toyota Prius gas-electric hybrid. It runs as drive-by-wire, e.g. the throttle position is read and a computer decides how much engine and how much electric motor to use.
And yes, some Prius drivers have reported the system can gets confused (not that often though, considering it has about 20 computer modules and 2 data busses).
You cannot reboot this car while driving, but you do NOT need to go to a dealer either. Usually the problem is when you are trying to get started. You just cycle the power once or twice and it will clear up if there is not a real fault. Even on real faults, you can usually keep driving or pull over and cycle the power.
Being a former automotive technician, I feel that I can comment fairly intelligently on this topic. There is such a world of difference in different makes and models of vehicles that there is no way that in 6 years all the hardware will be the same. For example, if you were to open the hood on a new Toyota and compare that with the underhood of a new Dodge, for example, there is nothing at all in common. Even the alloys used are completely different. The point being, there will always be differences on every concievable level between different car manufacturers.
Having formerly worked for an auto supplier designing microprocessor-based control modules for the Big Three, I have first-hand experience with microprocessor- and software-related warranty returns. I would not be the least bit surprised if this 30% figure is significantly on the high side. There was one vehicle model in particular that we got an enormous number of warranty returns on. We checked each and every one of them against the mechanic's report and found that less than 2% of the returned modules were actually defective, and the vast majority of those were unrelated to software or the microprocessor. My question, of course, is whether the ABI study would include ALL of those warranty returns, or just the ones that were actually defective.
The real problem was twofold: The module in question stored all of the fault codes for the entire vehicle, and it was located in a place where the mechanic could replace it in less than 10 minutes. As a result, the mechanic would see a fault code, match it up with an easy-to-replace module, and simply replace the module storing the fault code (never mind that the code was generated by, say, the engine controller). We would get the module back as a warranty return, and the mechanic would go on looking for the true source of the problem. I understand that certain expensive components are purposely made horribly difficult to replace so they won't be returned unless there truly is a problem with them.
This is not to say our software was perfect. We once discovered a bug that would reset the odometer if you took the right combination of actions. It was a very unusual combination of actions, but the bug was highly reproducible.
I am certain you were joking, but I have worked on *many* German cars, and what you did will clear the memory of any
fuel-injection ( or injection+ignition, e.g. Motronic ) computer.
This also applies to other cars. The memory used for the adaptive section of the box IS volatile, and can be dumped by
disconnecting the 12 volt power from it by disconnecting the
battery ground ( negative ) cable.
BTW, if you ever jump-start such a car, you can KILL the brain box by simply connecting jumper cables incorrectly.
A typical price for a brain box, new, is well over $1,000.
So, check those jumper cables carefully, before you touch them
to the battery or to ground, of your car.
A few years ago, while waiting for my plane to taxi off the runway, the captain told us we were delayed again since another piece of equipment in the tail section was sending a fault message to the cockpit and we couldn't take off until it stabilized.
After 30 minutes of tinkering without effect he announces, "Hang on. I'm going to reboot the sucker."
Well "the sucker" turned out to be the entire airplane. Every light and air-vent went dead for about 30 seconds and came on in an orderly fashion over the next few minutes. The problem was gone and we left in a few more minutes.
Not sure if you were joking there, but yes, BMW uses WinCE
-- If you try to fail and succeed, which have you done? - Uli's moose
That's an odd problem for sure, but it's pretty typical. I have a 1991 325i w/ Bavarian Autosport software. I usually cycle the power any time I change something that is part of the system (any sensors, filters, spark plugs, etc. etc.)
:-)
The car's original software is tuned to 87 octane fuel. The BavAuto chip tunes it to 91, but it's adaptive. When I reboot the computer, the car takes a while to re-learn stuff like how low it can idle before stalling. Every time I come to a stop, the tach drops and drops until the engine rumbles a little, then it gives it more gas. After doing this enough time it closes in on the optimum idle RPM. It's my standard practice to take the car out for a 30 minute "typical" drive after resetting the computer.
The E39 M5 is a GREAT car, I'm jealous!!