US 11,455,793 B2Granted

Robust object detection and classification using static-based cameras and events-based cameras

AssigneeIntel Corporation

InventorsLeobardo Campos Macias, Rafael De La Guardia Gonzalez, David Gomez Gutierrez, Anthony Kyung Guzman Leguel, Jose Ignacio Parra Vilchis

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

Techniques are disclosed to facilitate, in autonomous vehicles, the robust detection and classification of objects in a scene using a static sensors in conjunction with event-based sensors. A trained system architecture may be implemented, and the fusion of both sensors thus allows for the consideration of scenes with overexposure, scenes with underexposure, as well as scenes in which there is no movement. In doing so, the autonomous vehicle may detect and classify objects in conditions in which each sensor, if operating separately, would not otherwise be able to classify (or classify with high uncertainty) due to the sensing environment.

Claims21

Dependent claims

An object detection and classification system of an autonomous vehicle (AV), the object detection and classification system comprising:

one or more processors;
and
a memory configured to store instructions that, when executed by the one or more processors, cause the one or more processors to:
obtain a first set of images associated with a scene, the first set of images being encoded using static-based camera sensor data, each respective one of the first set of images being identified with a frame from among a plurality of frames;
for each respective frame from among the plurality of frames, obtain a second set of images associated with the scene, the second set of images including encoded information representing events occurring within each respective frame using event-based camera sensor data;
receive the first set of images and the second set of images via separate respective channel inputs, each one of the separate respective channel inputs being a dedicated channel input configured to provide, to a same input layer of a neural network, a different image from among the first set of images and the second set of images;
process (i) the first set of images including the static-based camera sensor data, and (ii) the encoded information included in the second set of images to determine a location and type of one or more objects included in the scene;
and
provide the location and type of the one or more objects included in the scene to a control system of the AV to perform one or more autonomous navigation tasks.

6 more elements

An autonomous vehicle (AV), comprising:

a media control unit configured to (i) obtain a first set of images associated with a scene, the first set of images being encoded using static-based camera sensor data, each respective one of the first set of images being identified with a frame from among a plurality of frames, and (ii) for each respective frame from among the plurality of frames, obtain a second set of images associated with the scene, the second set of images including encoded information representing events occurring within each respective frame using event-based camera sensor data;
and
an electronic control unit (ECU) configured to (i) receive the first set of images and the second set of images via separate respective channel inputs, each one of the separate respective channel inputs being a dedicated channel input configured to provide, to a same input layer of a neural network, a different image from among the first set of images and the second set of images, (ii) process the first set of images including the static-based camera sensor data, (iii) process the second set of images including the encoded information included in the second set of images, (iv) determine a location and type of one or more objects included in the scene, and (v) provide the location and type of the one or more objects included in the scene to a control system of the AV to perform one or more autonomous navigation tasks.

A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors of a control system associated with an autonomous vehicle (AV), cause the AV to:

obtain a first set of images associated with a scene, the first set of images being encoded using static-based camera sensor data, each respective one of the first set of images being identified with a frame from among a plurality of frames;
for each respective frame from among the plurality of frames, obtain a second set of images associated with the scene, the second set of images including encoded information representing events occurring within each respective frame using event-based camera sensor data;
receive the first set of images and the second set of images via separate respective channel inputs, each one of the separate respective channel inputs being a dedicated channel input configured to provide, to a same input layer of a neural network, a different image from among the first set of images and the second set of images;
process (i) the first set of images including the static-based camera sensor data, and (ii) the second set of images including the encoded information, to determine a location and type of one or more objects included in the scene;
and
provide the location and type of the one or more objects included in the scene to a control system of the AV to perform one or more autonomous navigation tasks.

3 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/829,203Filed 25 Mar 2020
Published: US 2020/0226377 A116 Jul 2020
Granted: 27 Sep 2022
Term ends: Not estimatedThe source records don’t support an estimate.

Classification12 codes

G06V 20/56
Context or environment of the image: exterior to a vehicle by using sensors mounted on the vehicle
G06F 18/214
Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation: generating training…
G06F 18/24
Analysing: classification techniques
G06F 18/251
Fusion techniques: of input or preprocessed data