Tesla patents data pipeline and deep learning system for autonomous driving

As Tesla is working toward deploying an autonomous driving system as soon as next year, the automaker is patenting a data pipeline and deep learning system that could help them develop it faster.

The system has been revealed in a new patent application by Tesla called ‘Data Pipeline and Deep Learning System for Autonomous Driving’.

Tesla’s data pipeline is likely one of the biggest in the industry, if not the biggest since it pulls data from a fleet of hundreds of thousands of vehicles equipped with a large suite of sensors.

In the application, Tesla explains the problem that its system is addressing:

“Deep learning systems used to implement autonomous driving typically rely on captured sensor data as input. In traditional learning systems, the captured sensor data is made compatible with a deep learning system by converting the captured data from a sensor format to a format compatible with the initial input layer of the learning system. This conversion may include compression and down-sampling that can reduce the signal fidelity of the original sensor data. Moreover, changing sensors may require a new conversion process. Therefore, there exists a need for a customized data pipeline that can maximize the signal information from the captured sensor data and provide a higher level of signal information to the deep learning network for deep learning analysis.”

Tesla describes its solution in the patent application:

“A data pipeline that extracts and provides sensor data as separate components to a deep learning network for autonomous driving is disclosed. In some embodiments, autonomous driving is implemented using a deep learning network and input data received from sensors. For example, sensors affixed to a vehicle provide real-time sensor data, such as vision, radar, and ultrasonic data, of the vehicle’s surrounding environment to a neural network for determining vehicle control responses. In some embodiments, the network is implemented using multiple layers. The sensor data is extracted into two or more different data components based on the signal information of the data. For example, feature and/or edge data may be extracted separate from global data such as global illumination data into different data components. The different data components retain the targeted relevant data, for example, data that will eventually be used to identify edges and other features by a deep learning network. In some embodiments, the different data components function as containers that store data highly relevant for identifying certain targeted features but do not themselves identify or detect the features. The different data components extract data to ensure accurate feature detection at an appropriate stage of a machine learning network. In some embodiments, the different data components may then be pre-processed to enhance the particular signal information that they contain. The data components can be compressed and/or down-sampled to increase resource and computational efficiency. The different data components are then provided to the deep learning system at different layers of the system.

The deep learning network is able to accurately identify and detect the features associated with the targeted data (e.g., edges, objects, etc.) of the data component using the signal information retained during extraction as input. For example, feature and edge data is provided to the first layer of the network and global data to a later layer of the network. By extracting different data components that each retain their respective targeted signal information, the network more efficiently processes the sensor data. Instead of receiving the sensor data as an initial input to the network, the network is provided with the most useful information at the most appropriate layer of the network. In some embodiments, a more complete version of the captured sensor data is analyzed by the network since the different data components can fully utilize the image resolution of their respective components for their intended purposes. For example, input for features and edges can utilize the entire resolution, bit range, and bit depth for feature and edge data whereas input for global illumination can utilize the entire resolution, bit range, and bit depth for global illumination data.”

Interestingly, two of the three engineers who are listed as inventors on the patent, Brijesh Tripathi and Timofey Uvarov, are not working at Tesla anymore.

Intel for Tripathi and Pony.Ai for Uvarov.

As for the third engineer, Evgeny Fainstain, it’s not clear if he is still at Tesla.

Here’s the full patent application from Tesla:

[scribd id=441072526 key=key-Mao5JoLi07Oi5n3bjffX mode=scroll]