Slashdot Mirror
Do Car Safety Problems Come From Outer Space?
Hugh Pickens writes "As electronic devices are made to perform more and more functions on smaller circuit chips, the systems become more sensitive and vulnerable to corruption from single event upsets. This is especially true of Toyota, which has led the auto industry in its widespread inclusion of electronic controls in the manufacture of their various car models. 'These circuit families store not just data, but their basic function electrically,' says Lloyd W. Massengill, director of engineering at the Vanderbilt Institute for Space and Defense Electronics at Vanderbilt University. 'In the unfortunate event of a particle flipping just the right bit, a circuit configured to carry out a benign action may be reprogrammed to carry out some unintended action.' Denise Chow writes in Live Science that some scientists are pointing to cosmic ray radiation as a plausible mechanism behind the sudden, unexplained acceleration reported to have occurred with the late model Toyotas."
"As the design of automobile systems continues to evolve from mechanical to electronic controls, relying more and more on various circuitry and chips, these electronic components may be vulnerable to being confounded by high-energy radiation writes Chow. Federal regulators were prompted to look into the possible role that cosmic rays played in Toyota's product recall fiasco after an anonymous tipster suggested the design of Toyota's microprocessors, software and memory chips could make them more vulnerable (PDF) to interference from radiation compared with other automakers. 'What's not known is what direction Toyota and other automakers are taking in terms of finding and correcting these issues,' says senior researcher Ewart Blackmore."
there should be some checksum that shouldn't add up. When a fault is detected, it should go to a backup program about safely shutting down the car.
Or how about a computer redundancy system where a group of computers that are all capable of controlling the car watch the behavior of the computer that is actually controlling the car. Through a voting system they could decide to hand the control of the car over to a another computer in the event that the controlling computer doesn't act in a way that was deemed safe. This way the car could continue to operate normally while signaling that there is a problem that needs to be addressed.
Look where all this talking got us, baby.
At the risk of sounding like a geezer, I remember back in the late 70's when this was a problem in early designs of mini-computers. Then we used to see single bits get flipped and crash computers from a variety of sources including cosmic radiation and alpha particles that came from the spontaneous decay of elements in the ceramic chip housings. More recently, when I purchased my 2005 Cadillac CTS it experienced a variety of problems similar to this when I would drive through a toll station that was equipped with RFID ID systems. Behaviours including sudden acceleration, engine stalling, indicator lights on the instrument panel going "crazy", On-Star calling for help when nothing was wrong, causing the driver's seat to suddenly drive forward to the steering wheel (making it really hard to steer), etc. At the time the only solution was to pull over, shut off the car, remove the key, open the door, wait for everything to shut down and then restart. After many frustrating weeks of "we can't duplicate the problem" it was discovered that the car had faulty shielding on one of the cables that makes up the in-car network. Once fixed the "gremlins" went away. The real crime here is that, because the problem can't be replicated on demand, Toyota is blaming the behaviour on attention seeking owners. This bizare response was recently repeated on the floor of Congress by one of Toyota's congressional tools. (I mean duly elected government representative.)
When I was working for NASA, on the NISN network, we'd get these weird router crashes for the old Cisco router located at (or very near) the South Pole in Antarctica. It was always a memory problem, and I'd always have to call someone to get them to powercycle the router. It irritated me to keep bothering those guys, so I opened a case with Cisco TAC.
The TAC guy sent a terse response, saying that particular crash was a "transient memory error" due to "alpha radiation or sun spots." That really pissed me off -- Cisco TAC just gave me a standard BOFH response! I escalated, and swung the NASA club around some, and finally got a senior engineer on the phone. "You said this router's at the South Pole, right? So that means it's at very high altitude, with very little ozone shielding, right?" "Umm, yeah." "Well there you go. There's a lot more radiation at that altitude than at sea level. Our stuff's only rated for sea level. See if they can .. I dunno, put a lead blanket over it or something."
I relayed the info to my contact at McMurdo, and he laughed and said he'd figure something out.
On a hunch, I checked the other two "high-altitude" routers we had, and sure enough, they both had a statistically higher failure rate for "transient memory errors".
http://unxmaal.com
I remember a news story from several years ago that even made the evening news. Someone had a Saturn car that they realized they couldn't afford and tried to return. The dealer wouldn't just take it back for a full refund, since it was now a used car.
Over the next few months, the driver had several "emergencies" with it, each time having it towed back to the dealership, where they couldn't find a problem. One in particular that was video taped by the police, the car was circling in a parking lot and the driver called 911. The insisted the car wouldn't stop. They told her to step on the brakes, use the emergency brake, throw it in neutral, shut it off, etc, etc, etc... She circled for something like 30 minutes. Finally they got her to open the drivers window, and an officer got in the middle of where it was circling. He ran for the side of the car, grabbed the wheel, and then turned off the key. The car (amazingly enough) came to a stop.
Of course, she claimed it wouldn't stop for her. There was all kinds of talk about lemon laws, and how Saturn vehicles weren't safe. She made a whole bunch of noise, and the dealership traded her car for another one. The problems persisted for her. Obviously Saturns were amazingly dangerous vehicles. Someone from the dealership (I think the owner) actually started driving her original car to work every day, to find out what the problem really was. He never had a problem.
Eventually, she was charged, I believe with reckless endangerment. Pretty much, she was driving dangerously, and endangered the officers who tried to help her.
I won't say that the mystery Toyota is driver error or a mechanical problem, but where the cases that have been in the news have massive parallels in other vehicles too, where drivers just did the wrong things.
A older lady in a Buick several years ago was pulling into the parking lot where I worked. I happened to be in the front of the store, and heard her tires squeal. She smashed into a parked car. That broke the parking pawl and sent the parked car across the parking lot into two other parked cars. One of those cars belonged to one of my coworkers, who wasn't exactly very happy that his car was totaled. I ran out to see if she was ok (once the cars stopped moving). She said "What happened?" I told her what she did. She was very insistent that she hit the brakes. I told her she spun the tires before hitting the first car. She said the other car must have done it. The driver of the other car was in the store at the time. At least everyone with wrecked cars had a good sense of humor about it, and no one was hurt. The funniest part was, her car was fine. There was absolutely no damage. It wasn't even scratched. The other three car were severely damaged though. Her insurance gave my coworker full book value on his car, even though it was a rusted piece of junk that barely ran. They were fully aware of it, they were just avoiding potential legal problems.
Serious? Seriousness is well above my pay grade.
My dad was an IBM CE (Customer Engineer) specialist on one of the models in the IBM System/360 mainframe range. He used to like telling the story about how he and another engineer were out on a customer's site trying to determine an intermittent fault. They would bring the machine up and sure enough there would be this glitch at precise intervals. They just couldn't figure out what was causing it. That was, until the other CE took a look out the window.
After a bit he said 'Tell me when it happens'. OK... '...now' my dad said. Then he said 'I'll tell you when the next one happens' and a few seconds later said '...now'. Which is exactly when it did glitch.
It turned out that the customer's DP center was situated close to an airport. The CE could see the radar dish revolve at the end of the runway. When it pointed straight at him was when the glitch occurred. Needless to say the computer room received some RF shielding.
Having heard all these stories really makes me wonder, i live in Belgium where cars with manual gear boxes are the common norm, and i've had my car accelerate like nuts once (pedal got stuck because of the floormat) i shifted to neutral, turned of the engine & used my momentum to get to the side of the road where i could dislodge the mat.
Are manual gearboxes that rare in the States?
I wouldn't say error, it was designed with parity protection only, so was incapable of correcting single bit errors, only detecting them. Hence, the reason for the crashes (i.e it detected a bit flip). If two bits were flipped you would never know.
I worked in the Sun front line call support during this time, and explaining this over and over to customers was somewhat painful. Never mind the years of mocking that still come from telling customers "it was a cosmic ray". Sun put massive effort into tracking, diagnosing and fixing this issue though. Some customers got versions of CPUs with "mirrored" SRAMs. Sad to say, I remember customers still getting errors with those.....
The US-III chips came out with end to end ECC protection, but the problems remained. In the end it turned out to be a host of socket mounting, pin contact and design specification issues that caused the errors, mostly solved by the time the 1200MHz CPUs were out. I wouldn't be surprised if it was something similar with the US-II.
As for Toyota, if they dont have end to end ECC they only have themselves to blame.
"If everybody is thinking alike, somebody isn't thinking" - Gen. George S. Patton