Tesla files patent for sourcing self-driving training data from its fleet

Tesla has filed a patent on how to source training data from its large fleet of customer vehicles in order to train its self-driving neural network.

The patent was filed for Tesla, but Andrej Karpathy, Tesla’s head of AI and Autopilot software, is named as the sole inventor on the application.

He describes the problem with collecting data for deep learning training in the application:

“Deep learning systems used for applications such as autonomous driving are developed by training a machine learning model. Typically, the performance of the deep learning system is limited at least in part by the quality of the training set used to train the model. In many instances, significant resources are invested in collecting, curating, and annotating the training data. The effort required to create the training set can be significant and is often tedious. Moreover, it is often difficult to collect data for particular use cases that a machine learning model needs improvement on.”

Tesla’s approach to developing a self-driving system is widely different than most other companies.

Whereas most other companies utilize relatively small fleets of test vehicles to collect data and test their systems, Tesla uses its fleet of hundreds of thousands of customer cars equipped with an array of sensors to collect road and driving data as well as test its self-driving system in “shadow mode”.

This data collected by the fleet is extremely valuable to Tesla in training its neural network for self-driving.

However, they have to be careful about what they collect and feed to the network.

Karpathy notes in the patent application:

“As machine learning models grow more complex, such as deeper neural networks, the necessity of large training data sets increases accordingly. These deeper neural networks may require a greater number of training examples as compared to shallower neural networks to ensure that their generalizability is high. For example, while a neural network may be trained to be highly accurate with respect to training data, the neural network may not generalize well to unseen future examples. In this example, the neural network may benefit from additional examples included in training data.”

Therefore, the engineer explains his patented method to classify potential training data at the source before transmitting it:

“An example method includes receiving sensor and applying a neural network to the sensor data. A trigger classifier is applied to an intermediate result of the neural network to determine a classifier score for the sensor data. Based at least in part on the classifier score, a determination is made whether to transmit via a computer network at least a portion of the sensor data. Upon a positive determination, the sensor data is transmitted and used to generate training data.”

Here are some drawings from the patent application explaining the method:

Here’s Tesla’s new patent application in full:

[scribd id=452883322 key=key-I7h96nwg1wM7XLeBOLGM mode=scroll]