Report That Tesla Autopilot Cuts Crashes By 40% Called 'Bogus' (arstechnica.com)

Remember when America's National Highway Traffic Safety Administration reported Tesla's Autopilot reduced crashes by 40%? Two years later the small research and consulting firm Quality Control Systems (QCS) finally obtained the underlying data -- and found flaws in the methodology "serious enough to completely discredit the 40 percent figure," reports Ars Technica, "which Tesla has cited multiple times over the last two years."
The majority of the vehicles in the Tesla data set suffered from missing data or other problems that made it impossible to say whether the activation of Autosteer increased or decreased the crash rate. But when QCS focused on 5,714 vehicles whose data didn't suffer from these problems, it found that the activation of Autosteer actually increased crash rates by 59 percent...

NHTSA undertook its study of Autopilot safety in the wake of the fatal crash of Tesla owner Josh Brown in 2016. Autopilot -- more specifically Tesla's lane-keeping function called Autosteer -- was active at the time of the crash, and Brown ignored multiple warnings to put his hands back on the wheel. Critics questioned whether Autopilot actually made Tesla owners less safe by encouraging them to pay less attention to the road. NHTSA's 2017 finding that Autosteer reduced crash rates by 40 percent seemed to put that concern to rest. When another Tesla customer, Walter Huang, died in an Autosteer-related crash last March, Tesla cited NHTSA's 40 percent figure in a blog post defending the technology. A few weeks later, Tesla CEO Elon Musk berated reporters for focusing on stories about crashes instead of touting the safety benefits of Autopilot....

[T]hese new findings are relevant to a larger debate about how the federal government oversees driver-assistance systems like Autopilot. By publishing that 40 percent figure, NHTSA conferred unwarranted legitimacy on Tesla's Autopilot technology. NHTSA then fought to prevent the public release of data that could help the public independently evaluate these findings, allowing Tesla to continue citing the figure for another year.... NHTSA fought QCS' FOIA request after Tesla indicated that the data was confidential and would cause Tesla competitive harm if it was released.
Last May the NHTSA finally clarified that their study "did not assess the effectiveness of this technology." Ars Technica also points out that the data focused on version 1 of Autopilot, "which Tesla hasn't sold since 2016."

Comments

Who are QCS?

[15Bit]

Not saying this is wrong, but QCS appears to be a husband and wife consulting outfit, looking at their website. Which raises the question - who checked over their research and OK'd this analysis of the statistics?

Re:Who are QCS?

[fluffernutter]

Also I don't believe they compared to other non-Autopilot cars in the same price range. They just compared to all other cars.

Re:Who are QCS?

[Entrope]

You don't have to take QCS's, or anyone else's, word about their work. Their report and the Ars article describe the statistically inappropriate decisions in the NHTSA study.

For example, the study did not have enough data to tell when Autosteer was enabled for about two thirds of the vehicles considered, so they approximated. That's not intently bad, but the approximation was that the 29,000 vehicles such vehicles had zero pre-Autosteer miles, which is very questionable. That mistake was seriously aggravated by including 18 accidents from these cars in the pre-Autosteer group, which obviously moves miles and/or accidents in a direction that made Autosteer look better.

Re:Who are QCS?

[110010001000]

Who checked the NHSTA analysis (or Tesla's). No one.

Re:Uh..

[Anonymous Coward]

I read Slashdot at the time, and that was pretty much the consensus back then.

Aside from people named Rei or had mod points.

Clearly Elon stole the Reality Distortion Field Steve Jobs pioneered.

Now we have

1. The NSA really is spying on everyone.
2. Bitcoin is an nonviable ponzi scheme.
3. Tesla autopilot is neither autopilot nor safe.

Anyone care to take bets on what is next? Mine is that solar and wind are cheaper than fossil fuel. We have heard it many times, but the numbers don't match quite yet.

Re:A new attempt against Tesla?

[fluffernutter]

Calling out radical flaws in a study that happened to say positive things about Tesla is considered "discrediting Tesla" now?

So what you're saying is...

[Anonymous Coward]

that a mere 18 cars in the original 'investigation' is enough to swing the results nearly 100 percentage points (+40 to -59).

Seems like yet another badly executed examination with a woefully undersized data set.

At least in this case it can be somewhat justified as the number of accidents involving Tesla cars is extremely small since there aren't that many of them out on the roads in their brief existence as a production vehicle.

After all, this whole thing started over one incident of a stupid driver misusing a driver assistance technology that was incorrectly called 'Autopilot' by the idiots at Tesla's marketing division.

Not surprisingly, when you make it sound like the car can drive itself, some of your braindead customers might actually believe you and operate the car as if it can drive itself -- who would've guessed?

XP

[fluffernutter]

Was the original study even adjusted for age? There will be less younger inexperienced drivers in a Tesla.

Autopilot

[110010001000]

Stop calling it AutoPilot. It is just drivers assist, just like every other car manufacturer puts in their high end vehicles. Now every jerk thinks they can fall asleep in their Tesla while cruising down the highway.

Re:Autopilot

[LynnwoodRooster]

Fall asleep at the wheel in your Tesla? That's stupid! I use the AutoPilot so I can peruse Slashdot and answ()&Y%*&P(Y*PPIHSFP(UBBIB:k3018yjnnu233^%%^*

Re:Still flawed

[whoever57]

This new report narrows down the pool of cars and crashes so much that there isn't enough data (there were not enough crashes) to get meaningful results from the analysis.