# US 12,216,475 B2 · Autonomous vehicle object classifier method and system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,216,475 B2
- Document ID: US12216475B2
- Status: Granted
- Kind: B2
- Application number: 17/411,894
- Filed: 2021-08-25
- Published: 2022-12-29
- Granted: 2025-02-04
- Earliest claimed: 2021-06-29
- Assignees: Honda Motor Co., Ltd.
- Applicants: Honda Motor Co., Ltd.
- Inventors: Jiachen Li; Haiming Gang; Hengbo Ma; Chiho Choi
- Web page: https://patentgrep.com/patent/US12216475B2

## CPC

- G05D 1/0221: {involving a learning process}
- B60W 50/08: Interaction between the driver and the control system
- B60W 60/0027: {using trajectory prediction for other traffic participants}
- G01S 7/4808: {Evaluating distance, position or velocity data}
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 17/89: for mapping or imaging
- G06F 18/24: Classification techniques
- G06F 18/2433: Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
- G06N 3/08: Learning methods
- G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]}
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G01S 17/931: of land vehicles
- G06F 18/253: {of extracted features}
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/806: {of extracted features}
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Object identification may be provided herein. A feature extractor may extract a first set of visual features, extract a second set of visual features, concatenate the first set of visual features, the second set of visual features, and a set of bounding box information, determine a number of object features and a global feature for a scene, and receive ego-vehicle feature information associated with an ego-vehicle. An object classifier may receive the number of object features, the global feature, and the ego-vehicle feature information, generate relational features with respect to relationships between each of the number of objects from the scene, and classify each of the number of objects from the scene based on the number of object features, the relational features, the global feature, the ego-vehicle feature information, and an intention of the ego-vehicle.

## Claims

### Claim 1

A system for object identification, comprising:
a feature extractor:
extracting a first set of visual features from a first image of a scene detected by a first sensor;
extracting a second set of visual features from a second image of the scene detected by a second sensor of a different sensor type than the first sensor;
concatenating the first set of visual features, the second set of visual features, and a set of bounding box information associated with the first image and the second image;
determining a number of object features associated with a corresponding number of objects other than an ego-vehicle from the scene and a global feature for the scene; and
receiving ego-vehicle feature information associated with the ego-vehicle; and
an object classifier:
receiving the number of object features, the global feature, and the ego-vehicle feature information;
generating relational features with respect to relationships between each of the number of objects from the scene, the relational features including a mutual influence between each of the number of objects from the scene;
classifying each of the number of objects from the scene as a binary classification of relevant or non-relevant to a behavior of the ego-vehicle, based on the number of object features, the relational features, the global feature, the ego-vehicle feature information, and an intention of the ego-vehicle; and
labeling each of the number of objects from the scene as relevant or non-relevant to the behavior of the ego-vehicle,
wherein the object classifier is trained utilizing semi-supervised learning including a labeled dataset and an unlabeled dataset, wherein the unlabeled dataset is annotated with pseudo labels generated from classifying each of the number of objects, the pseudo labels indicating each of the number of objects from the scene as relevant or non-relevant to the behavior of the ego-vehicle.

### Claim 2 (depends on 1)

The system for object identification for the ego-vehicle of claim 1, wherein the ego-vehicle feature information associated with the ego-vehicle includes a position, a velocity, or an acceleration associated with the ego-vehicle.

### Claim 3 (depends on 1)

The system for object identification for the ego-vehicle of claim 1, wherein the feature extractor sequence encodes the first set of visual features, the second set of visual features, and the set of bounding box information prior to concatenation.

### Claim 4 (depends on 1)

The system for object identification for the ego-vehicle of claim 1, wherein the generating relational features with respect to relationships between each of the number of objects from the scene is based on a fully-connected object relation graph, wherein each node corresponds to an object feature and each edge connecting two nodes represents a relationship between two objects associated with the two nodes.

### Claim 5 (depends on 1)

The system for object identification for the ego-vehicle of claim 1, comprising a task generator generating a task to be implemented via an autonomous controller and one or more vehicle systems using one or more of the number of object features associated with the corresponding number of objects from the scene classified as relevant to the behavior of the ego-vehicle, while discarding one or more of the number of object features associated with the corresponding number of objects from the scene classified as non-relevant to the behavior of the ego-vehicle.

### Claim 6 (depends on 5)

The system for object identification for the ego-vehicle of claim 5, wherein the task generator generates the task based on one or more of the number of object features associated with the corresponding number of objects from the scene classified as relevant to the behavior of the ego-vehicle, the ego-vehicle feature information, and the global feature, while discarding one or more of the number of object features associated with the corresponding number of objects from the scene classified as non-relevant to the behavior of the ego-vehicle.

### Claim 7 (depends on 5)

The system for object identification for the ego-vehicle of claim 5, wherein the task to be implemented includes an ego-vehicle action classifier and an ego-vehicle trajectory.

### Claim 8 (depends on 1)

The system for object identification for the ego-vehicle of claim 1, wherein the object classifier is trained utilizing supervised learning including a labeled dataset.

### Claim 9 (depends on 1)

The system for object identification for the ego-vehicle of claim 1, wherein the first sensor is an image capture sensor and the second sensor is a light detection and ranging (LiDAR) sensor.

### Claim 10

A computer-implemented method for object identification, comprising:
extracting a first set of visual features from a first image of a scene detected by a first sensor;
extracting a second set of visual features from a second image of the scene detected by a second sensor of a different sensor type than the first sensor;
concatenating the first set of visual features, the second set of visual features, and a set of bounding box information associated with the first image and the second image;
determining a number of object features associated with a corresponding number of objects other than an ego-vehicle from the scene and a global feature for the scene;
receiving ego-vehicle feature information associated with the ego-vehicle;
receiving the number of object features, the global feature, and the ego-vehicle feature information;
generating relational features with respect to relationships between each of the number of objects from the scene, the relational features including a mutual influence between each of the number of objects from the scene;
classifying, by an object classifier, each of the number of objects from the scene as a binary classification of relevant or non-relevant to a behavior of the ego-vehicle, based on the number of object features, the relational features, the global feature, the ego-vehicle feature information, and an intention of the ego-vehicle; and
labeling each of the number of objects from the scene as relevant or non-relevant to the behavior of the ego-vehicle,
wherein the method further comprises training the object classifier utilizing semi-supervised learning including a labeled dataset and an unlabeled dataset, wherein the unlabeled dataset is annotated with pseudo labels generated from classifying each of the number of objects, the pseudo labels indicating each of the number of objects from the scene as relevant or non-relevant to the behavior of the ego-vehicle.

### Claim 11 (depends on 10)

The method for object identification for the ego-vehicle of claim 10, wherein the ego-vehicle feature information associated with the ego-vehicle includes a position, a velocity, or an acceleration associated with the ego-vehicle.

### Claim 12 (depends on 10)

The method for object identification for the ego-vehicle of claim 10, comprising sequence encoding the first set of visual features, the second set of visual features, and the set of bounding box information prior to concatenation.

### Claim 13 (depends on 10)

The method for object identification for the ego-vehicle of claim 10, comprising generating the relational features based on a fully-connected object relation graph, wherein each node corresponds to an object feature and each edge connecting two nodes represents a relationship between two objects associated with the two nodes.

### Claim 14 (depends on 10)

The method for object identification for the ego-vehicle of claim 10, comprising generating a task to be implemented via an autonomous controller and one or more vehicle systems using one or more of the number of object features associated with the corresponding number of objects from the scene classified as relevant to the behavior of the ego-vehicle, while discarding one or more of the number of object features associated with the corresponding number of objects from the scene classified as non-relevant to the behavior of the ego-vehicle.

### Claim 15 (depends on 14)

The method for object identification for the ego-vehicle of claim 14, comprising generating the task based on one or more of the number of object features associated with the corresponding number of objects from the scene classified as relevant to the behavior of the ego-vehicle, the classifying each of the number of objects, the ego-vehicle feature information, and the global feature, while discarding one or more of the number of object features associated with the corresponding number of objects from the scene classified as non-relevant to the behavior of the ego-vehicle.

### Claim 16 (depends on 14)

The method for object identification for the ego-vehicle of claim 14, wherein the task to be implemented includes an ego-vehicle action classifier and an ego-vehicle trajectory.

### Claim 17 (depends on 10)

The method for object identification for the ego-vehicle of claim 10, comprising training the object classifier utilizing supervised learning including a labeled dataset.

### Claim 18

A system for object identification, comprising:
a feature extractor:
extracting a first set of visual features from a first image of a scene detected by a first sensor;
extracting a second set of visual features from a second image of the scene detected by a second sensor of a different sensor type than the first sensor;
concatenating the first set of visual features, the second set of visual features, and a set of bounding box information associated with the first image and the second image;
determining a number of object features associated with a corresponding number of objects other than an ego-vehicle from the scene and a global feature for the scene; and
receiving ego-vehicle feature information associated with the ego-vehicle;
an object classifier:
receiving the number of object features, the global feature, and the ego-vehicle feature information;
generating relational features with respect to relationships between each of the number of objects from the scene, the relational features including a mutual influence between each of the number of objects from the scene;
classifying each of the number of objects from the scene as a binary classification of relevant or non-relevant to a behavior of the ego-vehicle, based on the number of object features, the relational features, the global feature, the ego-vehicle feature information, and an intention of the ego-vehicle; and
labeling each of the number of objects from the scene as relevant or non-relevant to the behavior of the ego-vehicle;
a task generator generating a task to be implemented using one or more of the number of object features associated with the corresponding number of objects from the scene classified as relevant to the behavior of the ego-vehicle, while discarding one or more of the number of object features associated with the corresponding number of objects from the scene classified as non-relevant to the behavior of the ego-vehicle; and
an autonomous controller implementing the task by driving one or more vehicle systems to execute the task,
wherein the object classifier is trained utilizing semi-supervised learning including a labeled dataset and an unlabeled dataset, wherein the unlabeled dataset is annotated with pseudo labels generated from classifying each of the number of objects, the pseudo labels indicating each of the number of objects from the scene as relevant or non-relevant to the behavior of the ego-vehicle.