# US 2024/0199058 A1 · System and Method for Determining Object Intention Through Visual Attributes

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0199058 A1
- Document ID: US20240199058A1
- Status: Published application
- Kind: A1
- Application number: 18/591,413
- Filed: 2024-02-29
- Published: 2024-06-20
- Earliest claimed: 2018-06-15
- Assignees: Uatc, LLC
- Applicants: Uatc, LLC
- Inventors: Davi Eugenio Nascimento Frossard; Eric Randall Kee; Raquel Urtasun
- Web page: https://patentgrep.com/patent/US20240199058A1

## CPC

- B60W 50/14: Means for informing the driver, warning the driver or prompting a driver intervention
- B60W 60/0027: {using trajectory prediction for other traffic participants}
- B60W 60/00274: {considering possible movement changes}
- G05D 1/0246: {using a video camera in combination with image processing means}
- G05D 1/249: from positioning sensors located off-board the vehicle, e.g. from cameras
- G05D 1/81: Handing over between on-board automatic and on-board manual control
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V 20/584: {of vehicle lights or traffic lights}
- B60W 2050/0028: {Mathematical models, e.g. for simulation}
- B60W 2554/4045: Intention, e.g. lane change or imminent movement
- G06N 3/042: Knowledge-based neural networks; Logical representations of neural networks
- G06N 5/02: Knowledge representation; Symbolic representation
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06N 20/20: Ensemble learning
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods for determining object intentions through visual attributes are provided. A method can include determining, by a computing system, one or more regions of interest. The regions of interest can be associated with surrounding environment of a first vehicle. The method can include determining, by a computing system, spatial features and temporal features associated with the regions of interest. The spatial features can be indicative of a vehicle orientation associated with a vehicle of interest. The temporal features can be indicative of a semantic state associated with signal lights of the vehicle of interest. The method can include determining, by the computing system, a vehicle intention. The vehicle intention can be based on the spatial and temporal features. The method can include initiating, by the computing system, an action. The action can be based on the vehicle intention.

## Claims

### Claim at position 1

1-20. (canceled)

### Claim 21

A computer-implemented method, comprising:
obtaining sensor data associated with a surrounding environment of a first vehicle, wherein the sensor data comprises a sequence of image frames respectively corresponding to a plurality of time steps;
determining one or more regions of interest associated with the sensor data;
determining a model representation of an object associated with at least one of the one or more regions of interest;
determining one or more features indicative of one or more states associated with at least one signal of the object;
determining an intention indicative of a predicted future action of the object based on the model representation and the one or more features; and
initiating one or more actions based on the intention.

### Claim 22 (depends on 21)

The computer-implemented method of claim 21, wherein the object is a second vehicle.

### Claim 23 (depends on 22)

The computer-implemented method of claim 22, wherein the signal of the object is at least one signal light of the second vehicle, and wherein at least one of the one or more features is a temporal feature indicative of one or more states associated with the at least one signal light of the second vehicle.

### Claim 24 (depends on 21)

The computer-implemented method of claim 21, wherein the object is a bicycle.

### Claim 25 (depends on 24)

The computer-implemented method of claim 24, wherein the signal comprises a hand wave by an operator of the bicycle.

### Claim 26 (depends on 21)

The computer-implemented method of claim 21, wherein determining the intention indicative of the predicted future action of the object comprises determining the intention using a machine learned model, wherein the one or more features are input into the machine learned model.

### Claim 27 (depends on 21)

The computer-implemented method of claim 21, wherein the one or more regions of interest comprise one or more cropped image frames associated with the object.

### Claim 28 (depends on 27)

The computer-implemented method of claim 27, wherein the one or more cropped image frames comprise data indicative of the at least one signal of the object.

### Claim 29

A computing system comprising:
one or more processors;
one or more tangible, non-transitory, computer readable media that store instructions that are executable by the one or more processors to perform operations comprising:
obtaining sensor data associated with a surrounding environment of a first vehicle, wherein the sensor data comprises a sequence of image frames respectively corresponding to a plurality of time steps;
determining one or more regions of interest associated with the sensor data;
determining a model representation of an object associated with at least one of the one or more regions of interest;
determining one or more features indicative of one or more states associated with at least one signal of the object;
determining an intention indicative of a predicted future action of the object based on the model representation and the one or more features; and
initiating one or more actions based on the intention.

### Claim 30 (depends on 29)

The computing system of claim 29, wherein determining the one or more features comprise inputting a series of images of the one or more regions of interest into a machine learning model.

### Claim 31 (depends on 29)

The computing system of claim 29, wherein determining the intention associated with the object is inputting the one or more features into a machine learning model.

### Claim 32 (depends on 29)

The computing system of claim 29, wherein the model representation indicates an orientation of the object and wherein the orientation of the object is relative to an additional object within a surrounding environment of the first vehicle.

### Claim 33 (depends on 29)

The computing system of claim 29, wherein determining the one or more regions of interest comprise inputting the sequence of image frames into one or more machine learning models.

### Claim 34 (depends on 29)

The computing system of claim 29, wherein the features and the intention are respectively determined by different machine learned models of one or more machine learned models.

### Claim 35

A vehicle comprising, comprising:
one or more processors; and
one or more tangible, non-transitory, computer readable media that store instructions that are executable by the one or more processors to perform operations comprising:
obtaining sensor data associated with a surrounding environment of the vehicle, wherein the sensor data comprises a sequence of image frames respectively corresponding to one of a plurality of time steps;
determining one or more regions of interest associated with the sensor data;
determining a model representation of an object associated with at least one of the one or more regions of interest;
Serial Number: Applied for Herewith determining one or more features associated with at least one of the one or more regions of interest, wherein the one or more features are indicative of one or more states associated with at least one signal of the object;
determining an intention indicative of a predicted future action of the object based on the model representation and the one or more features; and
initiating one or more actions based on the intention.

### Claim 36 (depends on 35)

The vehicle of claim 35, wherein the vehicle is an autonomous vehicle.

### Claim 37 (depends on 35)

The vehicle of claim 35, wherein the operations further comprise:
generating a motion plan for the vehicle based on the intention.

### Claim 38 (depends on 37)

The vehicle of claim 37, wherein initiating the one or more actions is based on the motion plan.

### Claim 39 (depends on 35)

The vehicle of claim 35, wherein the vehicle further comprises one or more output devices, and wherein the operations further comprise:
providing, via the one or more output devices, data indicative of the intention to one or more operators of the vehicle.

### Claim 40 (depends on 35)

The vehicle of claim 35, wherein determining the intention associated with the object is inputting the one or more features into a machine learning model.
