Dashcam Video Shows Tesla Steering Toward Lane Divider - Again (arstechnica.com)

AmiMoJo shares a report from Ars Technica: The afternoon commute of Reddit user Beastpilot takes him past a stretch of Seattle-area freeway with a carpool lane exit on the left. Last year, in early April, the Tesla driver noticed that Autopilot on his Model X would sometimes pull to the left as the car approached the lane divider -- seemingly treating the space between the diverging lanes as a lane of its own. This was particularly alarming, because just days earlier, Tesla owner Walter Huang had died in a fiery crash after Autopilot steered his Model X into a concrete lane divider in a very similar junction in Mountain View, California.

Beastpilot made several attempts to notify Tesla of the problem but says he never got a response. Weeks later, Tesla pushed out an update that seemed to fix the problem. Then in October, it happened again. Weeks later, the problem resolved itself. This week, he posted dashcam footage showing the same thing happening a third time -- this time with a recently acquired Model 3. "The behavior of the system changes dramatically between software updates," Beastpilot told Ars. "Human nature is, 'if something's worked 100 times before, it's gonna work the 101st time.'" That can lull people into a false sense of security, with potentially deadly consequences.

Comments

Re:Not driving towards "lane divider"

[fluffernutter]

In my 30+ years of driving I have never seen that kind of signage or markers that are apparently used to dynamically close lanes at certain times

YES. That's what's hard about automated driving! Will we expect all construction companies everywhere to adopt universal signage and clean it and maintain it accurately? Not bloody likely!

Why do people think...

[110010001000]

....that autonomous driving is going to work? I mean, have you actually used software? Anything moderately complex has tons of bugs on it. And autonomous driving is extremely complex.

Re:Why do people think...

[Jeremi]

The only thing more hubristic than assuming something will definitely work is assuming something will never work.

Of course autonomous driving software will have bugs in it, and those bugs will lead to accidents. The status-quo alternative (biology-based driving software) also has bugs in it, which regularly leads to accidents.

The difference is that bugs in the autonomous driving software will eventually be diagnosed and fixed. Bugs in biological driving software, OTOH, will never be fixed, because every new person has to learn to drive from scratch; even if someone eventually becomes a flawless driver, sooner or later that person will die and replaced by another newbie, who will repeat the same newbie mistakes as everyone else. Lessons "learned" by software (and software designers) OTOH, can stay "learned" indefinitely, as long as they don't lose the source code.

Re:Why do people think...

[serviscope_minor]

All things are possible?

They don't. Well some nitwits do, but youre falling into the trap that because some things are physically impossible other things must be too. But when it comes to self driving cars you're pretty wide of the mark.

First, self driving cars aren't limited by physics like space travel is.

Secondly, you're ignoring the advancs in computer vision. Whether you believe deep learning is the key to strong AI or not (it isn't), or whether you believe it's 100% novel never seen before (it isn't), it has advanced practical comuter visison by a very large amount.

Thirdly, you're ignoring the history and advances. In the 1980s the first vision and lidar systems were made and could drive a car successfully on a closed track with no traffic. Cool demo but too early to be useful. Fsatforward to now with huge advances in all aspects of computer vision (both learning based and geometric), huge improvements in sensors, vast increases in computing power and you now have driverless cars which have indesutable successfully driven around in real cities with traffic. As in they might not be perfect or even good enough yet, but they do actually verifiably work.

You're basically stating that you kow that all future advances from about now will cease and self driving cars will stop just on the cusp of getting useful. Seems unlkely to me.

Finally you're ignoring the shockingly bad quality of the average human driver. A self driving car is not and will never be a human driver but worse. Because they always will and always have done a number of things vastly better than humans they can afford to do a fair few things considerably worse and still have better accident rates overall.

We are not going to get a self-driving car which is better than a well trained, fully alert human driver any time soon. That isn't going to happen. Can we get a self driving car with a lower accident rate than the average human? My money says a strong yes. Because while self driving cars will always foul up and drive into something no human ever would, they have other plus points:

They do not get tired. They do not get angry. They feel no impatience. They do not forget how to drive in conditions they haven't seen for a year. They are not distraced by their boss being an asshole at work. They don't text or use a mobile phone. They do not yell at their kids. They do not rush because they are late. They do not get close to the car in front. They do not race off the lights. They never try to show off or exhibit dominance. They don't daydream onboring journeys. They don't accelerate at pedestrians. They don't try to squeeze through gaps which are too small. They don't floor it to make the orange light. They won't pull out without checking mirrors. They wont forget to indicate. They won't realise they've misread the stanav then haul ass over three lanes with no warning to get to the turnoff. They won't drive the wrong way down a one way street because "it's only a short distance". They won't go twice the speed limit then slam on the brakes when they see a speed camera (and for bonus points blame other people for it). They wont speed in fog because they take lessons from the ravenous bugblatter beast of traal.

Now I, like 80% of the population rate myself as well above average as a driver. I don't need to be replaced with a self driving car. Those other idiots? A self driving car can be well below perfection and still be a lot better.

Re:More likely

[Rei]

He never reported the bug because he's apparently unaware of the in-vehicle bug reporting system, yet seems surprised that it's never been fixed.

Neural net vision systems train to their dataset. If your edge case is not in the dataset, it's not going to be learned. Self-driving vehicles without a driver at the wheel (Level 5) are not going to be viable for years because there's such a vast multitute of edge cases, and the only way to learn them is to collect an edge-case dataset. Until then, you're not getting beyond Level 3/4.

That's not to say that manually filing a bug report is the only way to trigger one. When there's a driver disengagement, that generates a sort of "mini-report". But a filed bug report contains a lot more information.

One of the things that I think a lot of people forget is that you're dealing with a data flood. Picture how much space and bandwidth it would take to record billions of miles of realtime, high framerate data from multiple cameras per vehicle, plus radar and ultrasonic sensors. Totally impractical - even just to transmit the full dataset for every edge case. As a result, there's some clever "data minimization" techniques used to keep the flood under control, where all "processed" data is transmitted (what it believes is where), but the raw data used to generate it is only transmitted at key moments (recognition ambiguity, changes in what's recognized, driver triggers, etc), and only for the objects of interest.

Also, as for this driver's particular case, I don't think it's hard to see what's going on. The car is correctly recognizing that the left lane is blocked, and correctly recognizing that the right lane is clear, but not realizing that you're not supposed to take the right lane, because of the arrow-sign in the left lane. So instead it's seeing the path that the other cars are taking as an exit, and trying to "remain on the same road" via the "clear" lane.

If I remember right, construction sign tagging was first spotted in shadow mode last fall. I'm not sure how much of that has since transitioned from shadow mode to live.

Re:I can't believe it

[ledow]

Duh. That's why anyone with a brain knows these things are deathtraps.

You can't debug a NN, not in any reasonable manner, certainly not one that you're constantly retraining and tweaking all the time. In this case, even providing heuristics ("Hey, there's a bridge near this GPS location, so don't think it's a wall" is literally what Tesla are putting into their software in some places because they can't train the behaviour out of the NN).

This has always been the concern of anyone that deals with such stuff since Tesla said they were using that technology.

You're basically training a black box on unknown criteria from limited test data, and then acting shocked when people say they don't understand how the black box works, can't predict what it will do, can't retrain or untrain it easily, and are surprised that even a million miles of road data aren't enough to let it drive safely across the entire world in perpetuity?

Re:Not driving towards "lane divider"

[thegarbz]

Will we expect all construction companies everywhere to adopt universal signage and clean it and maintain it accurately? Not bloody likely!

Huh? You Americans have a problem with standardising road and construction signage? To answer your question: yes, it is perfectly reasonable for a construction company to put in correct the correct procedures and equipment in order to maintain safety. That is literally a good chunk of the job of construction management.

Re:Not really AI at all

[thegarbz]

Tesla and just about everyone else in the "autonomous" driving game is using an Expert System.

Sorry but expert systems are not what does the image analysis. Go back to start. Do not collect $200.

Re:Not driving towards "lane divider"

[fluffernutter]

It doesn't work that way here. The construction unions are too powerful. They won't even take down 'reduce speed' signs at the end of the day. The city has to come and put up the official traffic markers which get left up for the entire duration, while the construction company uses their markers to route traffic around where they are digging or reconstructing. This ridiculousness is decades in the making, and a requirement for fancy new self-driving cars isn't about to change it.