Report That Tesla Autopilot Cuts Crashes By 40% Called 'Bogus' (arstechnica.com)
Remember when America's National Highway Traffic Safety Administration reported Tesla's Autopilot reduced crashes by 40%? Two years later the small research and consulting firm Quality Control Systems (QCS) finally obtained the underlying data -- and found flaws in the methodology "serious enough to completely discredit the 40 percent figure," reports Ars Technica, "which Tesla has cited multiple times over the last two years."
The majority of the vehicles in the Tesla data set suffered from missing data or other problems that made it impossible to say whether the activation of Autosteer increased or decreased the crash rate. But when QCS focused on 5,714 vehicles whose data didn't suffer from these problems, it found that the activation of Autosteer actually increased crash rates by 59 percent...
NHTSA undertook its study of Autopilot safety in the wake of the fatal crash of Tesla owner Josh Brown in 2016. Autopilot -- more specifically Tesla's lane-keeping function called Autosteer -- was active at the time of the crash, and Brown ignored multiple warnings to put his hands back on the wheel. Critics questioned whether Autopilot actually made Tesla owners less safe by encouraging them to pay less attention to the road. NHTSA's 2017 finding that Autosteer reduced crash rates by 40 percent seemed to put that concern to rest. When another Tesla customer, Walter Huang, died in an Autosteer-related crash last March, Tesla cited NHTSA's 40 percent figure in a blog post defending the technology. A few weeks later, Tesla CEO Elon Musk berated reporters for focusing on stories about crashes instead of touting the safety benefits of Autopilot....
[T]hese new findings are relevant to a larger debate about how the federal government oversees driver-assistance systems like Autopilot. By publishing that 40 percent figure, NHTSA conferred unwarranted legitimacy on Tesla's Autopilot technology. NHTSA then fought to prevent the public release of data that could help the public independently evaluate these findings, allowing Tesla to continue citing the figure for another year.... NHTSA fought QCS' FOIA request after Tesla indicated that the data was confidential and would cause Tesla competitive harm if it was released.
Last May the NHTSA finally clarified that their study "did not assess the effectiveness of this technology." Ars Technica also points out that the data focused on version 1 of Autopilot, "which Tesla hasn't sold since 2016."
The majority of the vehicles in the Tesla data set suffered from missing data or other problems that made it impossible to say whether the activation of Autosteer increased or decreased the crash rate. But when QCS focused on 5,714 vehicles whose data didn't suffer from these problems, it found that the activation of Autosteer actually increased crash rates by 59 percent...
NHTSA undertook its study of Autopilot safety in the wake of the fatal crash of Tesla owner Josh Brown in 2016. Autopilot -- more specifically Tesla's lane-keeping function called Autosteer -- was active at the time of the crash, and Brown ignored multiple warnings to put his hands back on the wheel. Critics questioned whether Autopilot actually made Tesla owners less safe by encouraging them to pay less attention to the road. NHTSA's 2017 finding that Autosteer reduced crash rates by 40 percent seemed to put that concern to rest. When another Tesla customer, Walter Huang, died in an Autosteer-related crash last March, Tesla cited NHTSA's 40 percent figure in a blog post defending the technology. A few weeks later, Tesla CEO Elon Musk berated reporters for focusing on stories about crashes instead of touting the safety benefits of Autopilot....
[T]hese new findings are relevant to a larger debate about how the federal government oversees driver-assistance systems like Autopilot. By publishing that 40 percent figure, NHTSA conferred unwarranted legitimacy on Tesla's Autopilot technology. NHTSA then fought to prevent the public release of data that could help the public independently evaluate these findings, allowing Tesla to continue citing the figure for another year.... NHTSA fought QCS' FOIA request after Tesla indicated that the data was confidential and would cause Tesla competitive harm if it was released.
Last May the NHTSA finally clarified that their study "did not assess the effectiveness of this technology." Ars Technica also points out that the data focused on version 1 of Autopilot, "which Tesla hasn't sold since 2016."
I read Slashdot at the time, and that was pretty much the consensus back then.
Laws are rules for the court, but merely a bottom bar to hit for life. Think beyond laws in your actions always.
Not saying this is wrong, but QCS appears to be a husband and wife consulting outfit, looking at their website. Which raises the question - who checked over their research and OK'd this analysis of the statistics?
funding secured, brah
(takes long drag off of a doobie)
" it found that the activation of Autosteer actually increased crash rates by 59 percent..."
Outstanding!
Let those tears flow, you cultist, fanboy pantywaists.
So the new number crunching by QCS ignores 18 crashes that happened before Autosteer was rolled out, because the total mileage of those vehicles prior to the crash was not known. Yet... those crashes most certainly occurred. In fact, it's quite possible that the crash itself resulted in enough destruction to the vehicle that the odometer could not be read and thus the exact mileage is not known and reported in the data.
With Autosteer enabled, more data is collected about the vehicle, so the exact mileage is known. QCS, in totally throwing out 18 crashes that definitely occurred to Teslas before Autosteer is as bad or worse than including them when the mileage was not known, again, because the crash could be the cause of the exact mileage not being known in a non-Autosteer state. If the crash was the cause of the missing data, and QCS uses missing data as filter to not include the crash in the statistics, then quite obviously they are going to see different results because of ignoring more than 20% of all the pre autosteer crashes that occurred.
Better known as 318230.
Calling out radical flaws in a study that happened to say positive things about Tesla is considered "discrediting Tesla" now?
Laws are rules for the court, but merely a bottom bar to hit for life. Think beyond laws in your actions always.
that a mere 18 cars in the original 'investigation' is enough to swing the results nearly 100 percentage points (+40 to -59).
Seems like yet another badly executed examination with a woefully undersized data set.
At least in this case it can be somewhat justified as the number of accidents involving Tesla cars is extremely small since there aren't that many of them out on the roads in their brief existence as a production vehicle.
After all, this whole thing started over one incident of a stupid driver misusing a driver assistance technology that was incorrectly called 'Autopilot' by the idiots at Tesla's marketing division.
Not surprisingly, when you make it sound like the car can drive itself, some of your braindead customers might actually believe you and operate the car as if it can drive itself -- who would've guessed?
Was the original study even adjusted for age? There will be less younger inexperienced drivers in a Tesla.
Laws are rules for the court, but merely a bottom bar to hit for life. Think beyond laws in your actions always.
Stop calling it AutoPilot. It is just drivers assist, just like every other car manufacturer puts in their high end vehicles. Now every jerk thinks they can fall asleep in their Tesla while cruising down the highway.
I accept Elon's alternative facts -- just like I accept Trumps -- without question!
The best cons are when the mark never knows he was conned.
They must be PEDOPHILES out to destroy Elons tripple A gold rating.
Well Musk is great at being a snake oil salesman and cooking the books on how successful Tesla is. But he's made bold unsubstantiated claims before so it's no surprise.
It turns out they were using those old pentium chips with the math co-processor error. It was supposed to be 0.45%!!!!
https://en.wikipedia.org/wiki/...
"Liberalism is a very noble idea, currently controlled by some very bad people. Be sure you do not get the two confused.
You can always find 1% which is in odds with the main body of facts.
;)
Nice sampling
Not only that but they also must be shorts and pedos.
My concern for stats on non-full-time autopilot is that it leaves the tricky, dangerous driving to a human.
Controlled-access highways in dry sunny weather are a lot easier to deal with than surface streets with frequent side entrances, in the rain, at night.
If you just naively compare accidents per mile of travel and don't limit the manual driving to equivalent road conditions, you're going to get a falsely high measurement of manual driving's accident rate.
Nissans anti-collision feels like 1960s tech, but it has saved me from a number of incidents. The insurance companies obviously know, but I donâ(TM)t think they are telling.
If you were right then insurance companies would charge young people the same as adults.
It never ceases to amaze me how so many /.ers will post completely random shit as fact when both common sense, well understood common knowledge or a quick search would show how wrong they are.
Younger inexperienced drivers cause more crashes. Period. End of story. Fact. You are wrong.
The guy's name is Randy Whitfield. Based on his resume he's at least got a statistical background and a paper or two to his name and he seems to have been completely silent since the last 80s.
The NHTSA study had it's flaws, but this guy needed to cut 90% of the data to get to his conclusion and the first 6 pages of his report seem to be a legal hit piece and a whine about the NHTSA not happily just handing over everything they've done and refusing to talk to him after he sued them (surprise).
But ad hominem aside what we really need is a *current* dataset opened up and analysed.
Autopilot would be great if it actively enforced driver attention. Then you get the best of both worlds. The car can perform mundane actions and react faster than any human when it needs to emergency brake etc. But the human is overseeing the car and road conditions and so able to prevent emergencies from developing in the first place. Sadly Tesla treated driver attention as an afterthought. Drivers are allowed to become inattentive and cannot intervene in time when the car does something dumb. The need to force attention should have been obvious from the beginning. Yet we still see photos of Tesla drivers apparently asleep at the wheel. The fact they can do this and the car does not detect it demonstrates that it is a dangerous system and still not fit for purpose.
Thank you AC, I couldn't bring myself to type the whole thing out.
Laws are rules for the court, but merely a bottom bar to hit for life. Think beyond laws in your actions always.
Autosteer feature is currently in Beta:
Autosteer is for use on highways that have a center divider and
clear lane markings, or where there is a car directly ahead to
follow. It should not be used on other kinds of roads or where the
highway has very sharp turns or lane markings that are absent,
faded, or ambiguous. Similar to the autopilot function in airplanes,
you need to maintain control and responsibility for your vehicle
while enjoying the convenience of Autosteer.
Do you want to enable Autosteer while it is in Beta?
[NO] [YES]
Period. End of story
Virtually everything can be distilled down to an oversimplification if you ask a numbers person... but you don't get what the fuck I'm talking about: Sure, older people are statistically safer drivers; after all, they're generally calmer (age) and more attentive (experience). Fuck's sake, that's obvious.
I didn't say anything about driving style; I was talking about skills.
Yes, there's a difference.
As for the numbers... well this story has already been debunked last week on electrek.
Short version: the one person company who made the analysis doesn't understand statistics that well and bases his conclusion on 1% of the total amount of autopilot driven distance...
I can only think of one reason for disinformation around this subject: stock price manipulation!
Y'all don't seem to understand what happened.
The guy wanted to check NHTSA. NHTSA stonewalled him on getting the raw data. He had to file FOIA to get data. When he got the data, he discovered that NHTSA did not have mileage figures for the pre-autopilot data. As a result, all accidents in that period do not have real mileage data to determine rates. The remaining data, perhaps not statistically significant (something like 1/3 of the data iirc) shows a much higher accident rate for post- compared to pre-.
Whether the guy is a one man consulting operation or a big outfit like Enron does not change that the NHTSA conclusions are not justified by the data they have. The guy's business is risk analysis, and he has real clients, so it's not out of the question that he actually knows something.
Sorry I don't partake, it makes me lazy.
***takes long drag from the joint, let's out a fake cough***
Calling out flaws in the original study is acceptable.
Publishing another flawed study by cutting out 90% of the data and cherry-picking the outcome you want is not acceptable.
"Go to CNN [for a] spell-checked, fact-checked summary" -- CmdrTaco