# US 2021/0342637 A1 · Generating ground truth for machine learning from time series elements

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342637 A1
- Document ID: US20210342637A1
- Status: Published application
- Kind: A1
- Application number: 17/301,965
- Filed: 2021-04-20
- Published: 2021-11-04
- Earliest claimed: 2019-02-01
- Assignees: Tesla, Inc.
- Applicants: Tesla, Inc.
- Inventors: Ashok Kumar Elluswamy; Matthew Bauch; Christopher Payne; Andrej Karpathy; Joseph Polin
- Web page: https://patentgrep.com/patent/US20210342637A1

## CPC

- G06N 3/08: Learning methods
- G05D 1/0221: {involving a learning process}
- G06F 18/28: Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road}
- G16Y 20/10: relating to the environment, e.g. temperature; relating to location
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G16Y: INFORMATION AND COMMUNICATION TECHNOLOGY SPECIALLY ADAPTED FOR THE INTERNET OF THINGS [IoT]

## Abstract

Sensor data, including a group of time series elements, is received. A training data set is determined, including by determining for at least a selected time series element in the group of time series elements a corresponding ground truth. The corresponding ground truth is based on a plurality of time series elements in the group of time series elements. A processor is used to train a machine learning model using the training dataset.

## Claims

### Claim at position 1

-20. (canceled)

### Claim 21

A method implemented by one or more processors, wherein the method comprises:
obtaining sensor data captured at respective times within a period of time;
determining a ground truth based on the sensor data, the ground truth comprising a three-dimensional feature associated with the sensor data; and
training a machine learning model using a training dataset comprising the determined ground truth and a portion of the sensor data captured at a particular time within the period of time, wherein the machine learning model is trained to output the ground truth based on an input of the portion of the sensor data.

### Claim 22 (depends on 21)

The method of claim 21, wherein the sensor data comprises a group of time series elements associated with respective times within the period of time.

### Claim 23 (depends on 21)

The method of claim 21, wherein the three-dimensional feature is formed from portions of the sensor data captured at respective times.

### Claim 24 (depends on 21)

The method of claim 21, wherein the three-dimensional feature reflects a lane line.

### Claim 25 (depends on 24)

The method of claim 24, wherein the sensor data comprises a plurality of images captured at respective times, and wherein the lane line is formed from different portions of the lane line as depicted in the plurality of images.

### Claim 26 (depends on 25)

the method of claim 25, wherein a portion of an individual image depicting a portion of the lane line is selected based on a measure associated with relevancy of the portion of the individual image with respect to remaining images depicting the portion of the lane line.

### Claim 27 (depends on 24)

The method of claim 24, wherein the three dimensional feature reflects a three-dimensional trajectory of the lane line.

### Claim 28 (depends on 21)

The method of claim 21, wherein the three-dimensional feature reflects a path associated with a vehicle.

### Claim 29 (depends on 28)

The method of claim 28, wherein the sensor data comprises a plurality of images captured at respective times, and wherein the machine learning model is trained to output the path based on an individual image of the vehicle.

### Claim 30 (depends on 28)

The method of claim 28, wherein the vehicle is an adjacent lane to a different vehicle which captured the sensor data.

### Claim 31 (depends on 21)

The method of claim 21, wherein the ground truth is determined based on odometry information associated with the sensor data.

### Claim 32 (depends on 21)

The method of claim 21, wherein the training dataset further comprises scene data describing a real-world environment around a vehicle which captured the sensor data.

### Claim 33 (depends on 21)

The method of claim 21, wherein the portion of the sensor data is selected based on it being within a threshold number of remaining portions of the sensor data as ordered according to respective time of capture.

### Claim 34

A system, comprising:
a processor; and
a memory coupled with the processor, wherein the memory is configured to provide the processor with instructions which when executed cause the processor to:
obtain sensor data captured at respective times within a period of time;
determine a ground truth based on the sensor data, the ground truth comprising a three-dimensional feature associated with the sensor data; and
train a machine learning model using a training dataset comprising the determined ground truth and a portion of the sensor data captured at a particular time within the period of time, wherein the machine learning model is trained to output the ground truth based on an input of the portion of the sensor data.

### Claim 35 (depends on 34)

The system of claim 34, wherein the sensor data comprises a group of time series elements associated with respective times within the period of time.

### Claim 36 (depends on 34)

The system of claim 34, wherein the three-dimensional feature reflects a lane line.

### Claim 37 (depends on 36)

The system of claim 36, wherein the sensor data comprises a plurality of images captured at respective times, and wherein the lane line is formed from different portions of the lane line as depicted in the plurality of images.

### Claim 38 (depends on 37)

The system of claim 37, wherein a portion of an individual image depicting a portion of the lane line is selected based on a measure associated with relevancy of the portion of the individual image with respect to remaining images depicting the portion of the lane line.

### Claim 39 (depends on 34)

The system of claim 34, wherein the portion of the sensor data is selected based on it being within a threshold of number remaining portions of the sensor data as ordered according to respective time of capture.

### Claim 40

A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions which when executed by a processor, cause the processor to:
obtain sensor data captured at respective times within a period of time;
determine a ground truth based on the sensor data, the ground truth comprising a three-dimensional feature associated with the sensor data; and
train a machine learning model using a training dataset comprising the determined ground truth and a portion of the sensor data captured at a particular time within the period of time, wherein the machine learning model is trained to output the ground truth based on an input of the portion of the sensor data.
