Slashdot Mirror
Sorry Elon Musk, There's No Clear Evidence Autopilot Saves Lives (arstechnica.com)
Timothy B. Lee writes for Ars Technica: A few days after the Mountain View crash, Tesla published a blog post acknowledging that Autopilot was active at the time of the crash. But the company argued that the technology improved safety overall, pointing to a 2017 report by the National Highway Traffic Safety Administration (NHTSA). "Over a year ago, our first iteration of Autopilot was found by the U.S. government to reduce crash rates by as much as 40 percent," the company wrote. It was the second time Tesla had cited that study in the context of the Mountain View crash -- another blog post three days earlier had made the same point. Unfortunately, there are some big problems with that finding. Indeed, the flaws are so significant that NHTSA put out a remarkable statement this week distancing itself from its own finding.
"NHTSA's safety defect investigation of MY2014-2016 Tesla Model S and Model X did not assess the effectiveness of this technology," the agency said in an email to Ars on Wednesday afternoon. "NHTSA performed this cursory comparison of the rates before and after installation of the feature to determine whether models equipped with Autosteer were associated with higher crash rates, which could have indicated that further investigation was necessary." Tesla has also claimed that its cars have a crash rate 3.7 times lower than average, but as we'll see there's little reason to think that has anything to do with Autopilot. This week, we've talked to several automotive safety experts, and none has been able to point us to clear evidence that Autopilot's semi-autonomous features improve safety. And that's why news sites like ours haven't written stories "about how autonomous cars are really safe." Maybe that will prove true in the future, but right now the data just isn't there. Musk has promised to publish regular safety reports in the future -- perhaps those will give us the data needed to establish whether Autopilot actually improves safety.
UPDATE (2/16/19): The study's underlying data reveals serious flaws in the methodology that undermine its credibility, according to new analysis from a research and consulting firm.
"NHTSA's safety defect investigation of MY2014-2016 Tesla Model S and Model X did not assess the effectiveness of this technology," the agency said in an email to Ars on Wednesday afternoon. "NHTSA performed this cursory comparison of the rates before and after installation of the feature to determine whether models equipped with Autosteer were associated with higher crash rates, which could have indicated that further investigation was necessary." Tesla has also claimed that its cars have a crash rate 3.7 times lower than average, but as we'll see there's little reason to think that has anything to do with Autopilot. This week, we've talked to several automotive safety experts, and none has been able to point us to clear evidence that Autopilot's semi-autonomous features improve safety. And that's why news sites like ours haven't written stories "about how autonomous cars are really safe." Maybe that will prove true in the future, but right now the data just isn't there. Musk has promised to publish regular safety reports in the future -- perhaps those will give us the data needed to establish whether Autopilot actually improves safety.
UPDATE (2/16/19): The study's underlying data reveals serious flaws in the methodology that undermine its credibility, according to new analysis from a research and consulting firm.
While Timothy normally does excellent articles, his reasoning and logic were severely flawed this time.
First off the NHTSA report focused on autosteer, not autobraking. Hence his attributing the reduction in accidents to autobraking is bizarre. What the NHTSA was disavowing was that they had not examined the entirety of Autopilot (which includes autosteer, autobraking, lane keeping, etc.). Timothy mistakenly thinks they were stating that they hadn't verified the effectiveness of autosteer installation in accident reduction (they didn't verify the actual usage, but drivers with autosteer installed use it about 50% of their driving time).
Secondly the Tesla's prior to the FSD update already had autobraking, so the 40% reduction in accidents after enabling FSD can't be attributed to the autobraking.