# US 12,118,461 B2 · Methods and systems for predicting dynamic object behavior

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,461 B2
- Document ID: US12118461B2
- Status: Granted
- Kind: B2
- Application number: 17/097,840
- Filed: 2020-11-13
- Published: 2022-05-19
- Granted: 2024-10-15
- Earliest claimed: 2020-11-13
- Assignees: Huawei Technologies Co., Ltd.
- Applicants: Amir Rasouli; Mohsen Rohani
- Inventors: Amir Rasouli; Mohsen Rohani
- Web page: https://patentgrep.com/patent/US12118461B2

## CPC

- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- B60W 30/0956: {the prediction being responsive to traffic or environmental parameters}
- B60W 60/0011: {involving control alternatives for a single driving scenario, e.g. planning several paths to avoid obstacles}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- B60W 2554/4046: Behavior, e.g. aggressive or erratic
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods and systems for predicting behavior of a dynamic object of interest in an environment of a vehicle are described. Time series feature data are received, representing features of objects in the environment, including a dynamic object of interest. The feature data are categorized into one of a plurality of defined object categories. Each categorized set of data is encoded into a respective categorical representation that represents temporal change of features within the respective defined object category. The categorical representations are combined into a single shared representation. A categorical interaction representation is generated based on the single shared representation that represents contributions of temporal change in each defined object category to a final time step of the shared representation. The categorical interaction representation together with data representing dynamics of the objects in the environment and data representing a state of the vehicle are used to generate predicted data representing a predicted future behavior of the dynamic object of interest.

## Claims

### Claim 1

A method for predicting behavior of a dynamic object of interest in an environment of a vehicle, the method comprising:
receiving a plurality of time series of feature data, each time series of feature data representing a respective feature of a plurality of objects in the environment over a plurality of time steps, the plurality of objects including a dynamic object of interest;
categorizing each time series of feature data into one of a plurality of defined object categories, each defined object category being defined by one or more shared characteristics of objects belonging to that defined object category, to obtain a categorized set of data for each defined object category, each categorized set of data containing one or more time series of feature data representing one or more objects belonging to the respective defined object category;
for each given categorized set of data belonging to a given defined object category, encoding, using a spatiotemporal encoder, the one or more time series of feature data belonging to the given categorized set of data into a single categorical representation, the single categorical representation being a time series of feature vectors representing temporal change of features within the given defined object category over the plurality of time steps, wherein a respective plurality of categorical representations is obtained for the plurality of defined object categories;
combining the plurality of categorical representations into a single shared representation that is a single time series of tensors that represents spatiotemporal changes in each of the plurality of defined object categories over the plurality of time steps;
generate a categorical interaction representation based on the single shared representation, the categorical interaction representation being a weighted representation of the single shared representation representing contributions of temporal change in each defined object category to a final time step of the single shared representation; and
generate predicted data representing a predicted future behavior of the dynamic object of interest based on the categorical interaction representation and data representing dynamics of the plurality of objects and data representing a state of the vehicle.

### Claim 2 (depends on 1)

The method of claim 1, wherein combining the categorical representations comprises concatenating the categorical representations according to time step, to generate the single shared representation.

### Claim 3 (depends on 1)

The method of claim 1, wherein encoding each categorized set of data comprises:
for the given categorized set of data belonging to the given defined object category, providing the one or more time series of feature data to a trained neural network to generate the time series of feature vectors as the single categorical representation for the given object category.

### Claim 4 (depends on 3)

The method of claim 3, wherein the trained neural network is a recursive neural network, a convolutional neural network, or a combined recursive and convolutional neural network.

### Claim 5 (depends on 1)

The method of claim 1, wherein at least one defined object category is specific to the dynamic object of interest.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
receiving a time series of sensor data generated by a sensor; and
preprocessing the time series of sensor data into one time series of feature data, the one time series of feature data being included in the received plurality of time series of feature data.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
providing the predicted data representing the predicted future behavior of the dynamic object of interest to a motion planning subsystem of the vehicle, to generate a planned path for the vehicle.

### Claim 8

A computing system for predicting behavior of a dynamic object of interest in an environment of a vehicle, the computing system comprising:
a processor system configured to execute instructions to cause an object behavior prediction subsystem of the computing system to:
receive a plurality of time series of feature data, each time series of feature data representing a respective feature of a plurality of objects in the environment over a plurality of time steps, the plurality of objects including a dynamic object of interest;
categorize each time series of feature data into one of a plurality of defined object categories, each defined object category being defined by one or more shared characteristics of objects belonging to that defined object category, to obtain a categorized set of data for each defined object category, each categorized set of data containing one or more time series of feature data representing one or more objects belonging to the respective defined object category;
for each given categorized set of data belonging to a given defined object category, encode, using a spatiotemporal encoder, the one or more time series of feature data belonging to the given categorized set of data into a single categorical representation, the single categorical representation being a time series of feature vectors representing temporal change of features within the given defined object category over the plurality of time steps, wherein a respective plurality of categorical representations is obtained for the plurality of defined object categories;
combine the plurality of categorical representations into a single shared representation that is a single time series of tensors that represents spatiotemporal changes in each of the plurality of defined object categories over the plurality of time steps;
generate a categorical interaction representation based on the single shared representation, the categorical interaction representation being a weighted vector representing contributions of temporal change in each defined object category to a final time step of the shared representation; and
generate predicted data based at least on the categorical interaction representation, data representing dynamics of the objects, and data representing a state of the vehicle, the predicted data representing a predicted future behavior of the dynamic object of interest.

### Claim 9 (depends on 8)

The computing system of claim 8, wherein the processing system is configured to execute instructions to combine the categorical representations by concatenating the categorical representations according to time step, to generate the single shared representation.

### Claim 10 (depends on 8)

The computing system of claim 8, wherein the processing system is configured to execute instructions to encode each categorized set of data by:
for the given categorized set of data belonging to the given defined object category, providing the one or more time series of feature data to a trained neural network to generate the time series of feature vectors as the single categorical representation for the given object category.

### Claim 11 (depends on 10)

The computing system of claim 10, wherein the trained neural network is a recursive neural network, a convolutional neural network, or a combined recursive and convolutional neural network.

### Claim 12 (depends on 8)

The computing system of claim 8, wherein at least one defined object category is specific to the dynamic object of interest.

### Claim 13 (depends on 8)

The computing system of claim 8, wherein the processing system is configured to execute instructions to cause the computing system to:
receive a time series of sensor data generated by a sensor; and
preprocessing the time series of sensor data into one time series of feature data, the one time series of feature data being included in the received plurality of time series of feature data.

### Claim 14 (depends on 8)

The computing system of claim 8, wherein the vehicle is an autonomous vehicle, wherein the computing system is implemented in the autonomous vehicle, and wherein the processing system is configured to execute instructions to cause a motion planning subsystem of the autonomous vehicle to:
receive the predicted data representing predicted future behavior of the dynamic object of interest as input; and
generate a planned path for the autonomous vehicle.

### Claim 15

A non-transitory computer readable medium comprising computer-executable instructions to implement an object behavior prediction subsystem to predict behavior of a dynamic object of interest in a sensed environment of a vehicle, wherein the instructions, when executed by a processing system of a computing system, cause the computing system to:
receive a plurality of time series of feature data, each time series of feature data representing a respective feature of a plurality of objects in an environment over a plurality of time steps, the plurality of objects including a dynamic object of interest;
categorize each time series of feature data into one of a plurality of defined object categories, each defined object category being defined by one or more shared characteristics of objects belonging to that defined object category, to obtain a categorized set of data for each defined object category, each categorized set of data containing one or more time series of feature data representing one or more objects belonging to the respective defined object category;
for each given categorized set of data belonging to a given defined object category, encode, using a spatiotemporal encoder, the one or more time series of feature data belonging to the given categorized set of data into a single categorical representation, the single categorical representation being a time series of feature vectors representing temporal change of features within the given defined object category over the plurality of time steps, wherein a respective plurality of categorical representations is obtained for the plurality of defined object categories;
combine the categorical representations into a single shared representation that is a single time series of tensors that represents spatiotemporal changes in each of the plurality of defined object categories over the plurality of time steps;
generate a categorical interaction representation based on the single shared representation, the categorical interaction representation being a weighted vector representing contributions of temporal change in each defined object category to a final time step of the shared representation; and
generate predicted data based on the categorical interaction representation, data representing dynamics of the plurality of objects, and data representing a state of the vehicle, the predicted data representing a predicted future behavior of the dynamic object of interest.

### Claim 16 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the instructions cause the computing system to combine the categorical representations by concatenating the categorical representations according to time step, to generate the single shared representation.

### Claim 17 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the instructions cause the computing system to encode each categorized set of data by:
for the given categorized set of data belonging to the given defined object category, providing the one or more time series of feature data to a trained neural network to generate the time series of feature vectors as the single categorical representation for the given object category.

### Claim 18 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein at least one defined object category is specific to the dynamic object of interest.

### Claim 19 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the instructions cause the computing system to:
receive a time series of sensor data generated by a sensor; and
preprocess the time series of sensor data into one time series of feature data, the one time series of feature data being included in the received plurality of time series of feature data.

### Claim 20 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the vehicle is an autonomous vehicle, and where the instructions cause the computing system to implement a motion planning subsystem of an autonomous vehicle to:
receive the predicted data representing the predicted future behavior of the dynamic object of interest as input; and
generate a planned path for the autonomous vehicle.