NHTSA released a report two years ago claiming that Tesla‘s vehicle crash rate dropped by 40% after the introduction of Autopilot.
Now a strange report is contesting the claim following a FOIA (freedom of information act) lawsuit to release the data, but Tesla defended the claim in a statement.
Tesla Autopilot Crash Rate Claim
It all goes back to January 2017 when NHTSA announced that it was closing its investigation of a fatal accident on Tesla Autopilot without finding any defect or issuing any recall.
In the final report, the federal agency also noted that Tesla’s crash rate was reduced by 40% after the Autopilot’s Autosteer feature was introduced based on their analysis of raw data supplied by Tesla.
Their analysis appeared to be fairly straightforward.
NHTSA’s Office of Defects Investigation (ODI) reviewed crash rate data from Tesla’s vehicles before and after the introduction of Autosteer:
ODI analyzed mileage and airbag deployment data supplied by Tesla for all MY 2014 through 2016 Model S and 2016 Model X vehicles equipped with the Autopilot Technology Package, either installed in the vehicle when sold or through an OTA update, to calculate crash rates by miles traveled prior to and after Autopilot installation.
They came to the conclusion that “the data show that the Tesla vehicles crash rate dropped by almost 40 percent after Autosteer installation.”
From 1.3 crashes per million miles before Autosteer to 0.8 crashes after Autosteer:
QCS Contesting NHTSA Report
Now the claim is being contested by Quality Control Systems Corp (QCS), which describes itself as a company providing “forensic statistical services and products”.
QCS filed a FOIA (freedom of information act) lawsuit against NHTSA for them to release everything they had about Tesla Autopilot.
The company eventually obtained the data and after analyzing it, they state that NHTSA couldn’t make that claim and that the data actually suggests that the crash rate went up after Tesla introduced Autopilot:
“After obtaining the formerly secret, underlying data through a lawsuit filed under the Freedom of Information Act (FOIA) against the U.S. Department of Transportation, we discovered that the actual mileage at the time the Autosteer software was installed appears to have been reported for fewer than half the vehicles NHTSA studied. For those vehicles that do have apparently exact measurements of exposure mileage both before and after the software’s installation, the change in crash rates associated with Autosteer is the opposite of that claimed by NHTSA – if these data are to be believed.”
In order to arrive at the opposite conclusion, they had to discard the vast majority of the vehicles equipped with Autopilot in the data set and used only 5,714 vehicles of the total 43,781 vehicles.
They claim that the data about the remainder of the vehicles couldn’t be used for a series of reasons that are difficult to follow.
QCS breaks down their entire analysis in the report, which you can find in full here:
[scribd id=399531084 key=key-GMfybzvNDXyaJlIJsGfF mode=scroll]
Tesla Defends Autopilot
Tesla disagrees with the analysis. Furthermore, they make the point that they used only “about 1% of the total mileage that Tesla vehicles have traveled to date with Autopilot engaged.”
According to the company, Tesla vehicles have now driven more than 10 billion miles, including more than 1 billion miles with Autopilot engaged.
Recently, Tesla also started publishing quarterly Vehicle Safety Report disclosing the crash rates between when drivers are using Autopilot versus when they aren’t.
The company’s latest quarterly safety report showed that accidents are more common on miles traveled without Autopilot.
Tesla sent us the following statement about the QCS report:
QCS’ analysis dismissed the data from all but 5,714 vehicles of the total 43,781 vehicles in the data set we provided to NHTSA back in 2016. And given the dramatic increase in the number of Tesla vehicles on the road, their analysis today represents about 0.5% of the total mileage that Tesla vehicles have traveled to date, and about 1% of the total mileage that Tesla vehicles have traveled to date with Autopilot engaged.
NHTSA’s original report did not only indicate that “Tesla vehicles crash rate dropped by almost 40 percent after Autosteer installation”, but the agency also concluded that it “did not identify any defects in the design or performance of the AEB or Autopilot systems” nor did it find “any incidents in which the systems did not perform as designed” (Page 10, last paragraph and Page 11, last paragraph, respectively). They also found that “the potential for driver misuse was evaluated as part of Tesla’s design process and solutions were tested, validated, and incorporated into the wide release of the product” (Page 10, first paragraph).
Our own vehicle safety data for Q3 and Q4, which includes data from roughly two billion miles driven in Tesla vehicles, shows that drivers using Autopilot were significantly less likely to be involved in an accident than those driving without using Autopilot.
Electrek’s Take
When I read the report last weekend, I thought it made no sense whatsoever.
Most of the data appeared to validate NHTSA’s analysis, but QCS was dismissing most of it for reasons I simply couldn’t understand.
Granted, I am not an expert in statistics, but it mostly looked to me like QCS couldn’t understand the data set properly.
I tried to look into QCS to see what kind of credibility they have and as far as I can tell, this is a single-man operation by a guy name Randy Whitfield, who is an epidemiologist.
I mean no disrespect, but if I have to trust someone to analyze the data set, I would choose NHTSA over this guy.
That’s why I didn’t post the report earlier, but it now looks like other media outlets are posting it and taking the report seriously so I thought I’d share my thoughts and Tesla’s response to the report to give a different perspective.
Ultimately, I don’t think it matter much either way since the data set has grown exponentially over the last two years and Autopilot went through several different iterations.
Data show that you are statistically less likely to have an accident while using Autopilot and I do believe that to be true as long as drivers are using the feature properly, which means to always stay alert and be ready to take control at all time.
It’s really as simple as that.
FTC: We use income earning auto affiliate links. More.
Comments