Replicating NHTSA's Extraordinary Safety Claims
about Tesla's Autopilot/Autosteer System
Quality Control Systems Corp. filed a Freedom of Information Act (FOIA) lawsuit on June 28, 2017 in federal District Court for the District of Columbia to obtain crash data withheld from the public by the National Highway Traffic Safety Administration (NHTSA). The data we are seeking involve crashes with airbag deployments and Tesla Incorporated's Autopilot driver assistance program. The data were analyzed by NHTSA as part of the Agency's investigation (PE16-007) into the Automatic Emergency Braking or Autopilot systems of Model Year 2014-2016 Tesla Model S and Model X vehicles.
Independent, scientific researchers need the data to assess the validity of the remarkable claim made by NHTSA that airbag deployments in Tesla vehicles dropped by almost 40 percent after the installation of a component of the Tesla's Autopilot system, Autosteer.
Our complaint against the U.S. Department of Transportation can be downloaded by clicking the image below:
The surprising claim by NHTSA of an extraordinary reduction in crash rates associated with the installation of Autosteer must be carefully considered in the context of the Agency's failure to allow public access to the underlying data. Such an important conclusion by the Agency should not be based on data that the government is withholding from researchers who want to examine NHTSA's results.
We seek to know if the methodology NHTSA used is scientifically valid and whether their results can be replicated. Other questions include whether the reduction in crash rates is actually due to Autosteer itself and whether the claimed crash reductions could be expected to continue over a longer period of time.
The surprising improvement in crash safety that NHTSA associates with Autosteer would be very welcomed if the dramatic safety claims prove to be scientifically sound. But it is concerning that the crash reductions are associated with the "installation" of Autosteer, rather than the actual use of Autosteer.
NHTSA's analysis is just as astonishing for the fact that it lacks the most basic information necessary for reaching well-founded conclusions about the claimed crash rate reductions. It is very remarkable that the published description of the Agency's findings do not meet long-established scientific standards that would allow for an assessment of statistical confidence intervals or of statistical significance. Even the numerators and the denominators of the calculated crash rates are AWOL.
NHTSA gave no reason at all to refuse to provide the data we requested through the Freedom of Information Act. If the safety benefits of Autosteer are as positive as the Agency claims, why wouldn't they want independent scientists to have the data in order to replicate these extraordinary results?
If you have questions about our research, please click here to send us an email.
http://www.quality-control.us/nhtsa_autopilot_safety_claims.html
Quality Control Systems Corp. Sues DOT for Tesla Data
Quality Control Systems (QCS) Corp. has filed a Freedom of Information Act (FOIA) lawsuit in the U.S. District Court for the District of Columbia in pursuit of Tesla airbag deployments data that the National Highway Traffic Safety Administration (NHTSA) has withheld from public view.
R. A. Whitfield, the company’s director, said that the company wanted to test the validity of claim made by NHTSA that airbag deployments in Tesla vehicles dropped by almost 40 percent after the installation of a component of the Tesla’s Autopilot system, Autosteer. NHTSA asserted this decrease in a report accompanying the Closing Resume of Preliminary Evaluation 16-007. The investigation was prompted by the May 2016 death of Joshua Brown, a Tesla enthusiast who was driving his Tesla Model S in Autopilot mode, when it crashed into an 18-wheel tractor-trailer truck that was turning left in front of it on US 27A, west of Williston, Florida. The report stated:
“ODI analyzed mileage and airbag deployment data supplied by Tesla for all MY2014 through 2016 Model S and 2016 Model X vehicles equipped with the Autopilot Technology Package, either installed in the vehicle when sold or through an OTA update, to calculate crash rates by miles travelled prior to and after Autopilot installation. Figure 11 shows the rates calculated by ODI for airbag deployment crashes in the subject Tesla vehicles before and after Autosteer installation. The data show that the Tesla vehicles crash rate dropped by almost 40 percent after Autosteer installation.”
Whitfield says he wants to know if the methodology NHTSA used is scientifically valid and whether their results can be replicated. Other questions include whether the reduction in crash rates is actually due to Autosteer itself and whether the claimed crash reductions could be expected to continue over a longer period of time.
“The surprising improvement in crash safety that NHTSA associates with Autosteer would be very welcomed if the dramatic safety claims prove to be scientifically sound. But it is concerning that the crash reductions are associated with the installation of Autosteer, rather than the actual use of Autosteer,” Whitfield says. “And NHTSA’s analysis is just as astonishing for the fact that it lacks the most basic, information necessary for reaching well-founded conclusions about the claimed crash rate reductions. It is very remarkable that the published description of the Agency’s findings do not meet long-established scientific standards that would allow for an assessment of statistical confidence intervals or of statistical significance. Even the numerators and the denominators of the calculated crash rates are AWOL.”
NHTSA’s Office of Defects Investigation opened the Tesla probe on June 28, 2016, focusing on whether the Automatic Emergency Braking (AEB) or Autopilot systems had functioned as designed, increasing the risk of a crash. It closed six months later with no defect finding, saying that the system performed as designed, and blaming Brown for the crash. Tesla’s four responses submitted to the public investigation file were almost wholly redacted. For more information about PE16007 and its lack of transparency, read Autonomous Vehicles, the
Safety Research & Strategies has long advocated for NHTSA transparency. For example, in February 2014, SRS submitted comments in advance of the agency finalizing its 2014 – 2018 Strategic Plan, highlighting its concern with NHTSA’s lack of transparency. SRS founder and President Sean Kane wrote: “Access to NHTSA’s investigations and data are increasingly difficult and expensive for the public and researchers as the agency assigns significant costs to provide information in response to FOIA requests. In some cases they have also refused to release information that should be public requiring FOIA litigation that has cost the Agency thousands of tax-payer dollars to settle.”
Since 2010, SRS has sued the Department of Transportation six times seeking public records on everything from child safety seats to unintended acceleration. All of these cases have been settled to our satisfaction. The four against NHTSA have ended with the agency agreeing to turn over more records and paying our fees, before a court judgement was rendered. You can read about our latest FOIA lawsuit here.
Whitfield says such an important conclusion by the agency should not be based on data that the government is withholding from researchers who want to examine NHTSA’s results.
“If the safety benefits of Autosteer are as positive as the Agency claims, why wouldn’t they want independent scientists to have the data in order to replicate these extraordinary results?” Whitfield asked.
http://www.safetyresearch.net/blog/articles/quality-control-systems-corp-sues-dot-tesla-data
