# US 12,223,428 B2 · Generating ground truth for machine learning from time series elements > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,428 B2 - Document ID: US12223428B2 - Status: Granted - Kind: B2 - Application number: 18/459,954 - Filed: 2023-09-01 - Published: 2024-02-29 - Granted: 2025-02-11 - Earliest claimed: 2019-02-01 - Assignees: Tesla, Inc. - Applicants: Tesla, Inc. - Inventors: Ashok Kumar Elluswamy; Matthew Bauch; Christopher Payne; Andrej Karpathy; Joseph Polin - Web page: https://patentgrep.com/patent/US12223428B2 ## CPC - G06N 3/08: Learning methods - G05D 1/0221: {involving a learning process} - G06F 18/28: Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road} - G16Y 20/10: relating to the environment, e.g. temperature; relating to location - G06N 3/04: Architecture, e.g. interconnection topology - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G16Y: INFORMATION AND COMMUNICATION TECHNOLOGY SPECIALLY ADAPTED FOR THE INTERNET OF THINGS [IoT] ## Abstract Sensor data, including a group of time series elements, is received. A training data set is determined, including by determining for at least a selected time series element in the group of time series elements a corresponding ground truth. The corresponding ground truth is based on a plurality of time series elements in the group of time series elements. A processor is used to train a machine learning model using the training dataset. ## Claims ### Claim 1 A method, comprising: receiving, by one or more processors, sensor data of a group of time series elements; determining, by the one or more processors, a training dataset, including by determining for at least a selected time series element in the group of time series elements a corresponding ground truth based on a plurality of time series elements in the group of time series elements; and training, by the one or more processors, using the training dataset, a machine learning model to output a predicted ground truth based on a single time series element, the training dataset comprising the selected time series element in the group of time series elements and the corresponding ground truth. ### Claim 2 (depends on 1) The method of claim 1, wherein determining the training dataset further comprises identifying, by the one or more processors, a relevant portion of the selected time series element of the group of time series elements, wherein the relevant portion includes one or more of a lane line, a traffic control signal, and an object. ### Claim 3 (depends on 1) The method of claim 1, wherein the corresponding ground truth is a three-dimensional representation of an attribute of the selected time series element of the group of time series elements. ### Claim 4 (depends on 3) The method of claim 3, wherein the three-dimensional representation represents a vehicle lane, a predicted path of a vehicle, an obstacle, a traffic control signal, a map feature, an object distance, or a drivable space. ### Claim 5 (depends on 1) The method of claim 1, wherein the sensor data of the group of time series elements comprises a plurality of images captured at respective times. ### Claim 6 (depends on 1) The method of claim 1, further comprising determining, by the one or more processors, the corresponding ground truth based on the sensor data. ### Claim 7 (depends on 6) The method of claim 6, wherein determining the corresponding ground truth comprises: identifying, by the one or more processors, for at least one time series element, a respective portion of the at least one time series element to form the corresponding ground truth, and generating, by the one or more processors, the corresponding ground truth based on the identified respective portion. ### Claim 8 (depends on 1) The method of claim 1, wherein determining the training dataset further comprises: selecting, by the one or more processors, an element of the group of time series elements; associating, by the one or more processors, the element of the group of time series elements with the corresponding ground truth; and including, by the one or more processors, the element and the associated corresponding ground truth in the training dataset. ### Claim 9 (depends on 1) The method of claim 1, further comprising inferring, by the one or more processors, a feature of the sensor data based on the training dataset, wherein the feature includes at least one of a vehicle lane, a drivable space, a pedestrian, a stationary vehicle, a moving vehicle, rain, hail, fog, a traffic light, a traffic sign, a street sign, and a traffic pattern. ### Claim 10 A system comprising: one or more processors; and a memory coupled with the one or more processors, wherein the memory is configured to provide the one or more processors with instructions which when executed cause the one or more processors to: receive sensor data of a group of time series elements; determine a training dataset, including by determining for at least a selected time series element in the group of time series elements a corresponding ground truth based on a plurality of time series elements in the group of time series elements; and train, using the training dataset, a machine learning model to output a predicted ground truth based on a single time series element, the training dataset comprising the selected time series element in the group of time series elements and the corresponding ground truth. ### Claim 11 (depends on 10) The system of claim 10, wherein determining the training dataset further comprises identifying, by the one or more processors, a relevant portion of the selected time series element of the group of time series elements, wherein the relevant portion includes one or more of a lane line, a traffic control signal, and an object. ### Claim 12 (depends on 10) The system of claim 10, wherein the corresponding ground truth is a three-dimensional representation of an attribute of the selected time series element of the group of time series elements. ### Claim 13 (depends on 12) The system of claim 12, wherein the three-dimensional representation represents a vehicle lane, a predicted path of a vehicle, an obstacle, a traffic control signal, a map feature, an object distance, or a drivable space. ### Claim 14 (depends on 10) The system of claim 10, wherein the sensor data of the group of time series elements comprises a plurality of images captured at respective times. ### Claim 15 (depends on 10) The system of claim 10, further comprising determining, by the one or more processors, the corresponding ground truth based on the sensor data. ### Claim 16 (depends on 15) The system of claim 15, wherein determining the corresponding ground truth comprises: identifying, by the one or more processors, for at least one time series element, a respective portion of the at least one time series element to form the corresponding ground truth, and generating, by the one or more processors, the corresponding ground truth based on the identified respective portion. ### Claim 17 (depends on 10) The system of claim 10, wherein determining the training dataset further comprises: selecting, by the one or more processors, an element of the group of time series elements; associating, by the one or more processors, the element of the group of time series elements with the corresponding ground truth; and including, by the one or more processors, the element and the associated corresponding ground truth in the training dataset. ### Claim 18 (depends on 10) The system of claim 10, further comprising inferring, by the one or more processors, a feature of the sensor data based on the training dataset, wherein the feature includes at least one of a vehicle lane, a drivable space, a pedestrian, a stationary vehicle, a moving vehicle, rain, hail, fog, a traffic light, a traffic sign, a street sign, and a traffic pattern. ### Claim 19 A non-transitory computer-readable storage medium comprising instructions which when executed by one or more processors, cause the one or more processors to: receive sensor data of a group of time series elements; determine a training dataset, including by determining for at least a selected time series element in the group of time series elements a corresponding ground truth based on a plurality of time series elements in the group of time series elements; and train, using the training dataset, a machine learning model to output a predicted ground truth based on a single time series element, the training dataset comprising the selected time series element in the group of time series elements and the corresponding ground truth. ### Claim 20 (depends on 19) The non-transitory computer-readable storage medium of claim 19, wherein the instructions further cause the one or more processors to: select an element of the group of time series elements; associate the element of the group of time series elements with the corresponding ground truth; and include the element and the associated corresponding ground truth in the training dataset.