US 9,734,455 B2Granted

Automated extraction of semantic information to enhance incremental mapping modifications for robotic vehicles

AssigneeZoox, Inc.

InventorsJesse Sol Levinson, Gabriel Thurston Sibley, Ashutosh Gajanan Rege

Life of this patent

Estimating the term…

Loading patent…

Abstract

Systems, methods and apparatus may be configured to implement automatic semantic classification of a detected object(s) disposed in a region of an environment external to an autonomous vehicle. The automatic semantic classification may include analyzing over a time period, patterns in a predicted behavior of the detected object(s) to infer a semantic classification of the detected object(s). Analysis may include processing of sensor data from the autonomous vehicle to generate heat maps indicative of a location of the detected object(s) in the region during the time period. Probabilistic statistical analysis may be applied to the sensor data to determine a confidence level in the inferred semantic classification. The inferred semantic classification may be applied to the detected object(s) when the confidence level exceeds a predetermined threshold value (e.g., greater than 50%).

Claims30

Dependent claims

A method, comprising:

receiving, at a computing system, first data sensed at a first time by a sensor system of an autonomous vehicle, the first data being representative of a first object of a plurality of objects on or proximate a road surface in a region in an environment the autonomous vehicle has autonomously navigated;
comparing the first data with reference data associated with a plurality of reference semantic classifications;
based at least in part on the comparing, determining that the first object does not match any of the plurality of reference semantic classifications;
identifying additional objects having additional object data similar to the first data, the first object and the additional objects comprising a subset of the objects;
receiving, at the computing system, second data sensed subsequent to the first time, the second data representing a behavior of at least one of the additional objects;
based at least in part on determining that the first object does not match any of the plurality of reference semantic classifications, determining, based on the first data and the second data, a probability that the subset of the objects conforms to a behavior;
generating, at the computing system, an inferred semantic classification associated with the subset of the objects when the probability indicates a pattern of objects in the subset of the objects conforming to the behavior;
associating the inferred semantic classification with the plurality of reference semantic classifications, the inferred semantic classification being different from each of the plurality of reference semantic classifications;
updating, at the computing system, map data associated with the environment to include information about the inferred semantic classification;
and
transmitting the updated map data to the autonomous vehicle and at least one additional autonomous vehicle.

8 more elements

A system comprising:

a bi-directional autonomous vehicle configured to drive forward in a first direction or drive forward in a substantially opposite second direction without turning around the bi-directional autonomous vehicle, the autonomous vehicle configured to drive autonomously on a roadway;
a plurality of sensors on the bi-directional autonomous vehicle configured to sense a plurality of objects on or proximate a roadway in an environment surrounding the bi-directional autonomous vehicle;
and
a computing system communicatively coupled to the bi-directional autonomous vehicle to receive data from the plurality of sensors, the computing system being programmed to:
determine first data for a first object of the plurality of objects, the first object being a moving object at a location in the environment;
compare the first data to reference semantic classifications data indicating object types and object behaviors at locations in the environment, the reference semantic classifications data being associated with one or more reference semantic classifications;
determine, based on the comparison, that the first object does not conform to any of the one or more reference semantic classifications;
determine, based at least in part on data acquired at different times, a pattern of behavior of additional moving objects at the location, each of the additional moving objects at the location having additional moving object data similar to the first data;
based at least in part on the determination that the first object does not conform to any of the one or more reference semantic classifications and based at least in part on the pattern of behavior of the additional moving objects at the location, associate an inferred semantic classification with moving objects at the location;
update the reference semantic classifications data to include the inferred semantic classification as an additional reference semantic classification;
and
update route data used to navigate the roadway based on the inferred semantic classification.

9 more elements

A system comprising:

a fleet of bi-directional autonomous vehicles, each of the bi-directional autonomous vehicles configured to drive forward in a first direction or drive forward in a substantially opposite second direction without turning around the autonomous vehicle;
a plurality of sensors on the bi-directional autonomous vehicles configured to sense a plurality of objects on or proximate a roadway upon which the bi-directional autonomous vehicles travel, in an environment surrounding the bi-directional autonomous vehicles;
one or more data stores storing a reference semantic classification, the reference semantic classification having reference data including data indicating an object type, data indicating an object behavior, and data indicating a location in the environment;
one or more data stores storing mapping information used by the fleet of bi-directional autonomous vehicles to navigate the environment;
and
a computing system communicatively coupled to the fleet of bi-directional autonomous vehicles to receive data from the plurality of sensors, the computing system being programmed to:
determine first object data associated with a first of the plurality of objects, the first object data including data indicating a first object type, data indicating a first object behavior, and data indicating a first object location;
compare the first object data to the reference data;
determine, based on the comparison, a difference between the reference data and the first object data, the difference comprising at least one of a difference between the data indicating the object type and the data indicating the first object type, a difference between the data indicating the object behavior and the data indicating the first object behavior, or a difference between the data indicating the location and the data indicating the first object location;
determine, based at least in part on first sensor data from a first of the bi-directional autonomous vehicles in the fleet of bi-directional autonomous vehicles and at least in part on second sensor data from a second of the bi-directional autonomous vehicles in the fleet of bi-directional autonomous vehicles, a pattern of behavior of additional moving objects at the first object location;
based at least in part on determining the difference between the reference data and the first object data and based at least in part on the pattern of behavior, associate an inferred semantic classification with the first object location or with objects at the first object location;
and
update the mapping information using the inferred semantic classification to indicate an object at the location.

10 more elements

Dependencies are inferred from the claim text.

Facts

Application: 14/932,940Filed 4 Nov 2015
Published: US 2017/0124476 A14 May 2017
Granted: 15 Aug 2017
Term ends: Not estimatedThe source records don’t support an estimate.

Classification115 codes

G06N 7/01
Computing arrangements based on specific mathematical models: probabilistic graphical models, e.g. probabilistic networks
B60L 3/0007
Electric devices on electrically-propelled vehicles for safety purposes; Monitoring operating variables, e.g. speed, deceleration or energy…:…
B60L 3/04
Electric devices on electrically-propelled vehicles for safety purposes; Monitoring operating variables, e.g. speed, deceleration or energy…:…
B60L 15/20
Methods, circuits, or devices for controlling the traction-motor speed of electrically-propelled vehicles: for control of the vehicle or its…