# US 12,111,386 B2 · Methods and systems for predicting a trajectory of an object

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,111,386 B2
- Document ID: US12111386B2
- Status: Granted
- Kind: B2
- Application number: 17/384,449
- Filed: 2021-07-23
- Published: 2022-01-27
- Granted: 2024-10-08
- Earliest claimed: 2020-07-24
- Assignees: Aptiv Technologies AG
- Applicants: Aptiv Technologies AG
- Inventors: Dominic Spata; Arne Grumpe; Mirko Meuter; Ido Freeman
- Web page: https://patentgrep.com/patent/US12111386B2

## CPC

- G01S 7/417: {involving the use of neural networks}
- G01S 13/58: Velocity or trajectory determination systems; Sense-of-movement determination systems
- G01S 13/06: Systems determining position data of a target
- G01S 13/726: {Multiple target tracking}
- G01S 13/931: of land vehicles
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A computer implemented method for predicting a trajectory of an object comprises the following steps carried out by computer hardware components: acquiring radar data of the object; determining first intermediate data based on the radar data based on a residual backbone using a recurrent component; determining second intermediate data based on the first intermediate data using a feature pyramid; and predicting the trajectory of the object based on the second intermediate data.

## Claims

### Claim 1

A method comprising:
acquiring, by computer hardware components, a plurality of frames of radar data of an object;
via a convolutional neural network, performing temporal fusion of features of the plurality of frames of radar data, wherein the convolutional neural network comprises a residual backbone, a feature pyramid and a regression head;
determining, using a recurrent component, first intermediate data based on the plurality of frames of radar data and the residual backbone;
determining, using the feature pyramid, second intermediate data based on the first intermediate data, wherein the second intermediate data comprises a plurality of features;
determining via the regression head a plurality of coefficients to time-continuous polynomial functions based on the plurality of features; and
predicting, based on the plurality of coefficients, a trajectory of the object.

### Claim 2 (depends on 1)

The method of claim 1, wherein predicting the trajectory of the object comprises determining a plurality of parameters, based on the second intermediate data, of the predicted trajectory of the object.

### Claim 3 (depends on 2)

The method of claim 2, wherein determining the plurality of parameters comprises:
determining polynomials of a pre-determined degree; and
determining the plurality of coefficients related to elements of a basis of a polynomial space of the polynomials of the pre-determined degree, the elements of the basis comprising monomials,
determining a first parameter, based on a position trajectory of the object, and a second parameter, based on an orientation of the object, and
wherein a velocity estimation, via differentiation, is provided.

### Claim 4 (depends on 1)

The method of claim 1, wherein the plurality of frames of radar data of the object comprise at least one of radar data cubes or radar point data.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
determining, using a classification head, fourth intermediate data based on the second intermediate data; and
predicting, based on the fourth intermediate data, the trajectory of the object.

### Claim 6 (depends on 1)

The method of claim 1, wherein the convolutional neural network comprises at least one of:
the residual backbone preceded by a recurrent layer stack; or
the residual backbone implemented as a recurrent residual backbone comprising a plurality of recurrent layers.

### Claim 7 (depends on 6)

The method of claim 6, wherein:
the convolutional neural network comprises the residual backbone implemented as the recurrent residual backbone and comprising the plurality of recurrent layers; and
the plurality of recurrent layers comprise a convolutional long short-term memory followed by convolution filters followed by a normalization processor.

### Claim 8 (depends on 6)

The method of claim 6, wherein:
the convolutional neural network comprises the residual backbone implemented as the recurrent residual backbone and comprising the plurality of recurrent layers; and
the plurality of recurrent layers comprise a convolution processor followed by a normalization processor followed by a rectified linear unit followed by a convolutional long short-term memory followed by another convolution processor followed by another normalization processor.

### Claim 9 (depends on 1)

The method of claim 1, wherein predicting the trajectory of the object comprises determining a tracklet for a plurality of time steps.

### Claim 10 (depends on 1)

The method of claim 1, wherein the recurrent component comprises a recurrent loop that is carried out once per time frame.

### Claim 11 (depends on 1)

The method of claim 1, wherein the recurrent component keeps hidden states between time frames.

### Claim 12 (depends on 1)

The method of claim 1, wherein the feature pyramid implements transposed strided convolutions.

### Claim 13 (depends on 1)

The method of claim 1, further comprising:
dividing a training data set into a plurality of subsets or retaining hidden states between optimization steps; and
training, based on the plurality of subsets divided from the training data or the hidden states retained between optimization steps, the computer hardware components to perform the method to predict the trajectory of the object.

### Claim 14 (depends on 1)

The method of claim 1, further comprising, via the regression head, providing the plurality of coefficients to the time-continuous polynomial functions that describe temporal evolution of bounding boxes in context of continuous polynomial path prediction.

### Claim 15 (depends on 1)

The method of claim 1, further comprising at least one of training or operating the convolutional neural network such that a recurrent loop is executed once each frame, while hidden states are kept between frames such that the convolutional neural network uses information from arbitrarily distant points in time and past sensor readings are not buffered and stacked to operate the convolutional neural network.

### Claim 16 (depends on 1)

The method of claim 1, further comprising determining the first intermediate data based on a time-sequence of two-dimensional multi-channel feature maps depicting a bird's eye view of an environment.

### Claim 17 (depends on 1)

The method of claim 1, further comprising:
evaluating the plurality of coefficients via a first polynomial function to provide a velocity time series;
evaluating the plurality of coefficients via a second polynomial function to provide a direction time series;
receiving size information from the regression head; and
providing the velocity time series, the direction time series and the size information to a regression loss processor.

### Claim 18

A system, the system comprising a plurality of computer hardware components configured to:
acquire a plurality of frames of radar data of an object;
via a convolutional neural network of the plurality of computer hardware components, perform temporal fusion of features of the plurality of frames of radar data, wherein the convolutional neural network comprises a residual backbone, a feature pyramid and a regression head;
determine, using a recurrent component, first intermediate data based on the plurality of frames of radar data based on the residual backbone;
determine, using the feature pyramid, second intermediate data based on the first intermediate data, wherein the second intermediate data comprises a plurality of features;
determine via the regression head a plurality of coefficients to time-continuous polynomial functions based on the plurality of features; and
predict, based on the plurality of coefficients, a trajectory of the object.

### Claim 19 (depends on 18)

The system of claim 18, wherein the plurality of computer hardware components are further configured to predict the trajectory of the object by determining a plurality of parameters of the predicted trajectory of the object based on the second intermediate data.

### Claim 20 (depends on 19)

The system of claim 19, wherein the plurality of computer hardware components are further configured to determine the plurality of parameters by:
determining polynomials of a pre-determined degree; and
determining the plurality of coefficients related to elements of a basis of a polynomial space of the polynomials of the pre-determined degree, the elements of the basis comprising monomials.

### Claim 21 (depends on 20)

The system of claim 20, further comprising:
determining a first parameter, based on a position trajectory of the object, and a second parameter, based on an orientation of the object; and
providing a velocity estimation via differentiation.

### Claim 22 (depends on 18)

The system of claim 18, wherein the system is implemented on a vehicle.

### Claim 23

A non-transitory computer readable medium comprising instructions, that when executed, configure at least one processor to:
acquire a plurality of frames of radar data of an object;
via a convolutional neural network, perform temporal fusion of features of the plurality of frames of radar data, wherein the convolutional neural network comprises a residual backbone, a feature pyramid and a regression head;
determine, using a recurrent component, first intermediate data based on the plurality of frames of radar data based on the residual backbone;
determine, using the feature pyramid, second intermediate data based on the first intermediate data, wherein the second intermediate data comprises a plurality of features;
determine via the regression head a plurality of coefficients to time-continuous polynomial functions based on the plurality of features; and
predict, based on the plurality of coefficients, a trajectory of the object.