US 11,380,108 B1Granted

Supplementing top-down predictions with image features

AssigneeZoox, Inc.

InventorsTianyi Cai, James William Vaisey Philbin, Kai Zhenyu Wang

Life of this patent

Estimating the term…

Loading patent…

Abstract

The described techniques relate to predicting object behavior based on top-down representations of an environment comprising top-down representations of image features in the environment. For example, a top-down representation may comprise a multi-channel image that includes semantic map information along with additional information for a target object and/or other objects in an environment. A top-down image feature representation may also be a multi-channel image that incorporates various tensors for different image features with channels of the multi-channel image, and may be generated directly from an input image. A prediction component can generate predictions of object behavior based at least in part on the top-down image feature representation, and in some cases, can generate predictions based on the top-down image feature representation together with the additional top-down representation.

Claims19

Dependent claims

A system comprising:

one or more processors;
and
one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:
receiving sensor data of an environment, the sensor data comprising image data;
determining an object based at least in part on the sensor data;
tracking a behavior of the object based at least in part on the sensor data;
inputting the image data into a first portion of a machine-learned model;
receiving, from the first portion of the machine-learned model, an image feature representation representing the object;
receiving a top-down representation, the top-down representation representing the object;
concatenate, as a concatenated representation, the image feature representation and the top-down feature representation, wherein the concatenated representation comprises a multi-channel representation including a first feature identified in the image representation and a second feature identified in the top-down feature representation;
inputting the concatenated representation into a second portion of the machine-learned model;
receiving, from the second portion of the machine-learned model, a predicted behavior of the object;
and
determining a difference between the predicted behavior of the object and the behavior of the object;
and
altering one or more parameters of one or more of the first portion or the second portion of the machine-learned model to minimize the difference.

13 more elements

A method comprising:

determining, based at least in part on sensor data from one or more sensors, a behavior of an object in an environment;
receiving a first image depicting the object in the environment, the first image captured at a first time;
receiving a second image depicting the object, the second image captured at a second time prior to the first time;
inputting at least a portion of the first image into a first portion of a machine-learned model;
inputting the second image into the first portion of the machine-learned model;
receiving, from the first portion of the machine-learned model, a first multi-channel representation;
receiving, from the first portion of the machine-learned model, a second multi-channel representation;
inputting the first multi-channel representation into a second portion of the machine-learned model;
inputting the second multi-channel representation into the second portion of the machine-learned model;
receiving, from the second portion of the machine-learned model, a predicted behavior of the object;
determining a difference between the predicted behavior of the object and the behavior of the object;
and
altering one or more parameters of the machine-learned model to minimize the difference.

10 more elements

One or more computer-readable media storing instructions that, when executed by one or more processors, perform operations comprising:

determining, based at least in part on sensor data from one or more sensors, a behavior of an object in an environment;
receiving a first image depicting the object in the environment;
inputting at least a portion of the first image into a first portion of a machine-learned model;
receiving a second image depicting the object, the second image being captured by a second camera;
receiving, from the first portion of the machine-learned model, a first multi-channel representation of the at least the portion of the image;
generating a top-down representation of the object in the environment based at least in part on the sensor data;
inputting the first multi-channel representation and the top-down representation into a second portion of the machine-learned model;
inputting the second image into a third portion of the machine-learned model;
receiving, from the third portion of the machine-learned model, a second multi-channel representation corresponding to the second image;
receiving, from the second portion of the machine-learned model, a predicted behavior of the object, wherein the predicted behavior is further based on the second multi-channel representation;
determining a difference between the predicted behavior of the object and the behavior of the object;
and
altering one or more parameters of one or more of the first portion or the second portion of the machine-learned model to minimize the difference.

10 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/586,646Filed 27 Sep 2019
Granted: 5 Jul 2022
Term ends: Not estimatedThe source records don’t support an estimate.

Classification17 codes

G06V 20/58
Exterior to a vehicle by using sensors mounted on the vehicle: recognition of moving objects or obstacles, e.g. vehicles or pedestrians;…
G05D 1/0088
Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots: characterized by the…
G05D 1/0221
With means for defining a desired trajectory: involving a learning process
G06F 18/217
Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation: validation; Performance…