Slashdot Mirror
Dashcam Video Shows Tesla Steering Toward Lane Divider - Again (arstechnica.com)
AmiMoJo shares a report from Ars Technica: The afternoon commute of Reddit user Beastpilot takes him past a stretch of Seattle-area freeway with a carpool lane exit on the left. Last year, in early April, the Tesla driver noticed that Autopilot on his Model X would sometimes pull to the left as the car approached the lane divider -- seemingly treating the space between the diverging lanes as a lane of its own. This was particularly alarming, because just days earlier, Tesla owner Walter Huang had died in a fiery crash after Autopilot steered his Model X into a concrete lane divider in a very similar junction in Mountain View, California.
Beastpilot made several attempts to notify Tesla of the problem but says he never got a response. Weeks later, Tesla pushed out an update that seemed to fix the problem. Then in October, it happened again. Weeks later, the problem resolved itself. This week, he posted dashcam footage showing the same thing happening a third time -- this time with a recently acquired Model 3. "The behavior of the system changes dramatically between software updates," Beastpilot told Ars. "Human nature is, 'if something's worked 100 times before, it's gonna work the 101st time.'" That can lull people into a false sense of security, with potentially deadly consequences.
Beastpilot made several attempts to notify Tesla of the problem but says he never got a response. Weeks later, Tesla pushed out an update that seemed to fix the problem. Then in October, it happened again. Weeks later, the problem resolved itself. This week, he posted dashcam footage showing the same thing happening a third time -- this time with a recently acquired Model 3. "The behavior of the system changes dramatically between software updates," Beastpilot told Ars. "Human nature is, 'if something's worked 100 times before, it's gonna work the 101st time.'" That can lull people into a false sense of security, with potentially deadly consequences.
He isn't replicating the situation consistently and it's never been fixed.
Are they sharing the same autopilot dev team?
Tesla's autopilot automatically takes aim at anything the camera doesn't recognize and the Boeing 737-Max autopilot automatically takes a 90 degrees plunge to the ground the moment something abnormal happens.
There are parallels here..
It's clear in the video the the Telsa is trying to take the left lane that has that strange signage showing it is closed. When the driver steers back to the right at that point it is heading towards the divider, but the car is trying to take that lane that goes to the left of the barrier. That's different than "the car is trying to steer into the lane divider".
In my 30+ years of driving I have never seen that kind of signage or markers that are apparently used to dynamically close lanes at certain times. I would wonder what I was seeing myself the first time I encountered that.
It looks like two things are going on:
1) The visual system of the Tesla does not understand that signage meaning a lane / offramp has been closed.
2) The GPS routing shows that is a viable route when it is somehow only intermittently open.
Better known as 318230.
This must be the autopilot in the Boeing 737 Max 8!
Laws are rules for the court, but merely a bottom bar to hit for life. Think beyond laws in your actions always.
Not funny. Not in my garage.
....that autonomous driving is going to work? I mean, have you actually used software? Anything moderately complex has tons of bugs on it. And autonomous driving is extremely complex.
No kidding. It's like Adobe Flash updates of the past. Except now the bug fixes introduce new people killing issues.
its AI is just wanting to ends it misery sooner than later.. phoning home...HAL, Joshua, D.a.r.r.y.l.... no one answers.
Except those idiots are also putting OUR lives in the hands of Tesla engineers. The NHTSB needs to put a stop to this already before more people are killed.
Duh. That's why anyone with a brain knows these things are deathtraps.
You can't debug a NN, not in any reasonable manner, certainly not one that you're constantly retraining and tweaking all the time. In this case, even providing heuristics ("Hey, there's a bridge near this GPS location, so don't think it's a wall" is literally what Tesla are putting into their software in some places because they can't train the behaviour out of the NN).
This has always been the concern of anyone that deals with such stuff since Tesla said they were using that technology.
You're basically training a black box on unknown criteria from limited test data, and then acting shocked when people say they don't understand how the black box works, can't predict what it will do, can't retrain or untrain it easily, and are surprised that even a million miles of road data aren't enough to let it drive safely across the entire world in perpetuity?
Back in 94/95 a friend and I went to a Progfest in LA. My navigator was poring over his paper maps trying to figure out where I should get off. I was in the middle lane of a 5-6 lane freeway, ready to go in either direction at a moment's notice. Keep in mind this is at 70 MPH, surrounded by other cars doing 70. And Ken was a pretty good navigator.
About the time he said "shit!", I said "shi!t" as the freeway split into 2. 2-3 lanes going left, 2-3 lanes going right, and I was on an offramp straight down the middle. Turned out to be the offramp I wanted.
I remember the year(s) because I bought a brand spankin new car in '94, and still considered it brand new for a year after.
AI will be the end of us.
They also can't drive into sunlight, in snow, in rain, in fog, in construction areas, in places with potholes, in places with faded reflective paint, in places where other drivers are idiots, can't discern basic optical illusions, can't figure out what road heat mirages are, can't read text on some road signs...basically self-driving cars are one giant lie and only work under extremely controlled conditions and the technology to drive in even 90% of worldwide driving conditions won't be available for 50 years.
The video clearly shows that the Tesla was in the Ravenna section of Seattle, which is reasonably nice. It was simply trying to avoid heading further south into the lower-class area known as the University District.
#DeleteChrome
I know. I for one will only buy cars where I have a lower chance of survival and less safety features. None of this guinea pig stuff.
Tesla and just about everyone else in the "autonomous" driving game is using an Expert System.
Sorry but expert systems are not what does the image analysis. Go back to start. Do not collect $200.
While substantial reward$s fathomable if able to iffy autonomous transport , it is monumental undertaking. There are simpler incremental safety and efficiency tech solutions that could help in near term such as drive recorders , smart roads that can share road/traffic conditions , monitor dangerous drivers etc... Smarter roads can help autonomous driving. But since financial risk / rewards dispersed less investment. Still transportation getting better. The ride hailing app investors are subsidizing a transition to Sharing idle assets. Car sharing/pooling will get better as cost of ownership management improves. Meanwhile pumping venture money into Engineering still helpful for economy, since they will spin off into other areas that might seem more practical.
"Human nature is, 'if something's worked 100 times before, it's gonna work the 101st time.'" That can lull people into a false sense of security, with potentially deadly consequences.
You got that right.
When you are dealing with AI, and it gets retrained, it MUST be retested fully.
And it appears that this edge-case is not being tested.
I prefer the "u" in honour as it seems to be missing these days.
Time for
Ralph Nader to write an Unsafe at Any Speed 2 auto ride of death.
There are hundreds of millions of cars on the road with no sensors at all other than two human eyes. I’m not sure why a biological neural net can drive on two human eyes but a digital neural net needs 75 times of radar.
When you are dealing with AI, and it gets retrained, it MUST be retested fully.
Not quite right. You are assuming that the machine learning technique involved suffers from Catastrophic Forgetting upon re-training. This was a problem back in the early days of machine learning, but any modern AI engineer and researcher knows of this problem and is or will be implementing solutions.
When a human learns to fly a Cessna, we get a pilot's license. When we get a type certificate to fly an Airbus after learning to fly the Cessna, we don't forget how to fly the Cessna and need re-training in the Cessna.... unless we don't fly a Cessna for a long time. Humans are engineered to forget, which is fundamentally important for being human, but not at all important for a control system.
I am glad to see this distinction has been discussed. Because of the difficulty in fixing this problem it is probably due to a combination of control loop in the inflexible decision logic and error in the camera based image recognition system. Musk needs to admit error in relying on fantasy technology and add an array of $100 LIDAR sensors to break these problems. Hell, why not use a few hundred of them for redundancy? Nothing beats detecting the actual real world as a backup, and it should simplify a lot of the stupidly complex things they are doing that increase cost to the luxury level for what is barely an economy car in build quality.
I'm a pilot. I fly a plane with an autopilot. I also drive a Tesla with their "autopilot".
The very expensive aircraft autopilot flies great. I can be hands-off the controls for extended periods of time, read a book, browse Facebook (hurrah for GoGo :), etc. Do I? Hell no! An aircraft autopilot has no clue what other aircraft are doing. TCAS might see another nearby aircraft, maybe it won't. I keep my hands on or near the controls, I look out the window, and I scan the instruments - all the time. Which is pretty much what I do in the Tesla. The big difference is that the Tesla actually does a pretty decent job of reacting to other cars. Odd lane markings and construction zones do freak it out from time to time. I have had the Tesla alert me to an unsafe traffic or road condition and tell me to take over - in a flurry of beeps and on-screen alerts. Freaks me out every time. I wish the autopilot in the airplane would do that - instead it just shuts off, throws a warning light if I'm lucky, and the plane wanders off somewhere in the sky until I pull head of my my ass. I probably hand-fly the airplane more than I hand-drive the Tesla - on cross country trips. Taxiing around on the ground is a bit like driving a Tesla to the grocery store - an annoying fact of life to tolerate only until I get where I belong - out on the road, or up in the air, where the massively automated systems not only make my life easier, they make it safer as well.
You people bitching about how dangerous the Tesla autopilot is are just spoiled, bitchy little meat bags of self-loading cargo. You have no concept of automation, risk, and capability, you see the autopilot and cry that it's not perfect. You all need to fly from LA to NYC in a Ford Trimotor, or drive between them in a model T. Keep a spare set of points and a condenser in the glovebox. The magneto on the Trimotor's radial engines probably uses the same points as the Model T. Make sure you can change the points and gap them in the middle of nowhere, because that's where they'll fail. You'll be flying for about 20 hours, and you'll make about 8 stops for fuel and maintenance. The Model T will take a wee bit longer, at least 60 hours, with modern roads, unless you have to stop and fix the engine. A model 3 can make that drive in 50 hours, and you won't have to change the points once.
Japan actually banned them from doing tests on customers. Tesla cars in Japan have old versions of the software because the regulator realized it was incredibly dumb to do constant over-the-air updates that alter the behaviour of the car and which have not been certified or properly tested.
const int one = 65536; (Silvermoon, Texture.cs)
SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
Driving on the road with one of these present will present and unlimited capacity for chaos because if something unexpected (or unprogrammed) happens, the car will do something unexpected. And that could be dangerous to everyone around.
Good point! This is why self driving cars will never work because they do unexpected things and humans never do. It must have been a self driving car I saw over a decade ago which suddenly hauled it over 3 lanes to the middle, pulled a u turn and then floored it back in the other direction. And that time bender got trashed on WD-40 at 2 in the afternoon and kept swerved his beaten up F150 between lanes.
Bloody robots.
SJW n. One who posts facts.
....pay me to buy these kind of cars.
A system that produces an audible warning if the driver drifts away from the middle of the lane makes some degree of sense. I think if you need that, the correct response is to find an exit and take a break; so I guess these have a purpose as a tired driver alert system.
What is the purpose of automatically staying in the lane? The driver is still obliged to pay attention. There doesn't seem to be any more cognitive load to actually turning the steering wheel. All this does is remove that warning that you might be too tired.
Also we don't know how our brains work but we still use our brains.
If you go back in history, many technologies have been deployed without the knowledge of how the technology works.
Just look at how medicines are created, if a compound was shown to improve a patient's outcome and had an acceptable level of side-effects then doctors can use the medicine. The doctors don't need to know how the medicine works to deploy treatments that scientifically are known to work.
Therefore, black box Neural Nets can be treated just like a human driver by presenting data inputs and evaluating the outputs. The innards of the system don't need to be understood to demonstrate their capabilities in the real-world. As long as the Neural Net has a higher probability of success than a human driver then the Neural Net will reduce deaths. The Neutral Net does not need to be 100% safe, it just needs to be better than a human.
In the UK at least, councils monitor where people are killed on the roads and where possible, modifications to road layouts are done to reduce the likelihood of future fatal crashes at those locations. The same processes will happen for autonomous vehicle fatal crashes. Designs of roads will gradually evolve over the years as has always been the case in the UK.