US 12,117,838 B1 · Object tracking and entity resolution

Abstract

Described herein is a system for tracking objects and performing dynamic entity resolution using image data. For example, the system may build an environment map and populate the map with objects present in the environment. As the devices move about the environment it may capture image data and, based on its position and/or configuration of its components, may determine updated locations of objects that move in the environment. Upon receiving a query from a user, based on the location of the objects relative to the device/user, the system can interpret gestures and voice commands to infer which object is specified by the voice command. To build the environment map, the system performs object detection to generate bounding boxes associated with an object, then clusters the bounding boxes into a three-dimensional (3D) object associated with 3D coordinates. As the system tracks the object using the 3D coordinates while maintaining two-dimensional (2D) information (e.g., bounding boxes and other features), the system can use existing 2D models to process objects in 3D.

Claims21

Dependent claims

A computer-implemented method, the method comprising:

receiving environment data representing a three-dimensional map of an environment;
moving, by a device, to a first location in the environment;
determining a configuration of a mechanical component of the device, the mechanical component comprising a camera;
determining first position data representing the first location and the configuration;
receiving, from the camera, first image data representing the environment;
performing object detection using the first image data to determine an object;
based on determining the object, determining first stored data corresponding to a previous location of the object;
determining, using the environment data and the first position data, a first direction in which the camera is directed while the device is at the first location;
determining a first bounding box corresponding to a portion of the first image data representing the object;
based at least in part on the first position data and the first direction, determining the first bounding box corresponds to a second location;
determining second position data corresponding to the second location;
and
determining second stored data corresponding to the object being located at the second location.

10 more elements

A system comprising:

at least one processor;
and
memory including instructions operable to be executed by the at least one processor to cause the system to:
move a device to a first position in an environment;
determine first position data representing the first position, wherein the first position data comprises data representing a configuration of a mechanical component of the device, the mechanical component including a first image capture component;
receive, from at least the first image capture component of the device, first image data of the environment;
perform object detection using the first image data to determine an object;
based on determining the object, determine first stored data corresponding to a previous position of the object;
determine, using at least the first position data and the first image data, second position data corresponding to a current position of the object;
and
determine second stored data corresponding to the current position of the object.

8 more elements

Dependencies are inferred from the claim text.

Object tracking and entity resolution

Life of this patent

Object tracking and entity resolution

Life of this patent

Abstract

Claims21