# US 12,117,838 B1 · Object tracking and entity resolution

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,117,838 B1
- Document ID: US12117838B1
- Status: Granted
- Kind: B1
- Application number: 17/218,621
- Filed: 2021-03-31
- Granted: 2024-10-15
- Earliest claimed: 2021-03-31
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: Gunnar Atli Sigurdsson; Robinson Piramuthu; Gokhan Tur
- Web page: https://patentgrep.com/patent/US12117838B1

## CPC

- G05D 1/0219: {ensuring the processing of the whole working surface}
- G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)}
- G05D 1/0251: {extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision (stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593)}
- G05D 1/0274: {using mapping information stored in a memory device (navigation using map-matching G01C21/30)}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06T 7/73: using feature-based methods
- G06V 20/20: in augmented reality scenes
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06V 40/193: {Preprocessing; Feature extraction}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- G10L 15/1807: {using prosody or stress}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L 2015/226: {using non-speech characteristics}
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Described herein is a system for tracking objects and performing dynamic entity resolution using image data. For example, the system may build an environment map and populate the map with objects present in the environment. As the devices move about the environment it may capture image data and, based on its position and/or configuration of its components, may determine updated locations of objects that move in the environment. Upon receiving a query from a user, based on the location of the objects relative to the device/user, the system can interpret gestures and voice commands to infer which object is specified by the voice command. To build the environment map, the system performs object detection to generate bounding boxes associated with an object, then clusters the bounding boxes into a three-dimensional (3D) object associated with 3D coordinates. As the system tracks the object using the 3D coordinates while maintaining two-dimensional (2D) information (e.g., bounding boxes and other features), the system can use existing 2D models to process objects in 3D.

## Claims

### Claim 1

A computer-implemented method, the method comprising:
receiving environment data representing a three-dimensional map of an environment;
moving, by a device, to a first location in the environment;
determining a configuration of a mechanical component of the device, the mechanical component comprising a camera;
determining first position data representing the first location and the configuration;
receiving, from the camera, first image data representing the environment;
performing object detection using the first image data to determine an object;
based on determining the object, determining first stored data corresponding to a previous location of the object;
determining, using the environment data and the first position data, a first direction in which the camera is directed while the device is at the first location;
determining a first bounding box corresponding to a portion of the first image data representing the object;
based at least in part on the first position data and the first direction, determining the first bounding box corresponds to a second location;
determining second position data corresponding to the second location; and
determining second stored data corresponding to the object being located at the second location.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
receiving, by the device, first audio data representing speech of a user;
performing speech processing on the first audio data to generate speech processing data;
determining that the speech processing data indicates the object; and
causing an action to be performed based at least in part on the second stored data.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, further comprising:
receiving, by the device, second image data including a second representation of the environment;
processing the second image data to determine at least one of a first direction in which a face of the user is oriented or a second direction in which the user is pointing; and
determining that at least one of the first direction or the second direction is associated with the second stored data,
wherein determining that the speech processing data indicates the object is based at least in part on determining that at least one of the first direction or the second direction is associated with the second stored data.

### Claim 4

A computer-implemented method, the method comprising:
moving, by a device, to a first position in an environment;
determining first position data representing the first position;
receiving, from at least a first image capture component of the device, first image data of the environment;
performing object detection using the first image data to determine an object;
based on determining the object, determining first stored data corresponding to a previous position of the object;
determining, using at least the first position data and the first image data, second position data corresponding to a current position of the object;
determining second stored data corresponding to the current position of the object, receiving, by the device after determining the second stored data, first audio data representing speech of a first user;
performing speech processing on the first audio data to generate speech processing data;
determining that the speech processing data indicates the object; and
causing an action to be performed based at least in part on the second stored data.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein the first position data comprises data representing a configuration of a mechanical component of the device, the mechanical component including the first image capture component.

### Claim 6 (depends on 4)

The computer-implemented method of claim 4, further comprising including time data with the second stored data.

### Claim 7 (depends on 4)

The computer-implemented method of claim 4, further comprising:
performing user recognition using one or more of second image data or second audio data to determine a second user who interacted with the object; and
including user data with the second stored data.

### Claim 8 (depends on 4)

The computer-implemented method of claim 4, further comprising:
receiving, by the device, second image data including a second representation of the environment;
processing the second image data to determine at least one of a first direction in which a face of the first user is oriented or a second direction in which the first user is pointing; and
determining that at least one of the first direction or the second direction is associated with the second stored data,
wherein determining that the speech processing data indicates the object is based at least in part on determining that at least one of the first direction or the second direction is associated with the second stored data.

### Claim 9 (depends on 4)

The computer-implemented method of claim 4, wherein causing the action to be performed further comprises:
determining, using the speech processing data, output data indicating the previous position of the object;
performing text-to-speech (TTS) processing using the output data to determine output audio data; and
causing the device to playback the output audio data.

### Claim 10 (depends on 4)

The computer-implemented method of claim 4, further comprising:
receiving environment data representing a three-dimensional map of the environment;
determining, using the environment data, a first direction in which the first image capture component is directed while the device is at the first position;
determining a first bounding box corresponding to a portion of the first image data representing the object; and
based at least in part on the position data and the first direction, determining that the first bounding box corresponds to the second position data.

### Claim 11 (depends on 4)

The computer-implemented method of claim 4, further comprising:
determining a first bounding box corresponding to a portion of the first image data representing the object,
wherein the second stored data includes the second position data and data corresponding to the first bounding box.

### Claim 12 (depends on 4)

The computer-implemented method of claim 4, wherein causing the action to be performed further comprises:
determining, using the speech processing data and the second stored data, output data indicating the current position of the object;
performing text-to-speech (TTS) processing using the output data to determine output audio data; and
causing the device to playback the output audio data.

### Claim 13 (depends on 4)

The computer-implemented method of claim 4, wherein causing the action to be performed further comprises:
determining, using the speech processing data and the second stored data, a first location associated with the current position of the object; and
moving, by the device, to the first location.

### Claim 14

A system comprising:
at least one processor; and
memory including instructions operable to be executed by the at least one processor to cause the system to:
move a device to a first position in an environment;
determine first position data representing the first position, wherein the first position data comprises data representing a configuration of a mechanical component of the device, the mechanical component including a first image capture component;
receive, from at least the first image capture component of the device, first image data of the environment;
perform object detection using the first image data to determine an object;
based on determining the object, determine first stored data corresponding to a previous position of the object;
determine, using at least the first position data and the first image data, second position data corresponding to a current position of the object; and
determine second stored data corresponding to the current position of the object.

### Claim 15 (depends on 14)

The system of claim 14, wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to include time data with the second stored data.

### Claim 16 (depends on 14)

The system of claim 14, wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
perform user recognition using one or more of second image data or audio data to determine a user who interacted with the object; and
include user data with the second stored data.

### Claim 17 (depends on 14)

The system of claim 14, wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
receive, by the device after determining the second stored data, first audio data representing speech of a user;
perform speech processing on the first audio data to generate speech processing data;
determine that the speech processing data indicates the object; and
cause an action to be performed based at least in part on the second stored data.

### Claim 18 (depends on 17)

The system of claim 17, wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
receive, by the device, second image data including a second representation of the environment;
process the second image data to determine at least one of a first direction in which a face of the user is oriented or a second direction in which the user is pointing; and
determine that at least one of the first direction or the second direction is associated with the second stored data,
wherein the instructions that cause the system to determine that the speech processing data indicates the object are based at least in part on a determination that at least one of the first direction or the second direction is associated with the second stored data.

### Claim 19 (depends on 17)

The system of claim 17, wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
determine, using the speech processing data, output data indicating the previous position of the object;
perform text-to-speech (TTS) processing using the output data to determine output audio data; and
cause the device to playback the output audio data.

### Claim 20 (depends on 14)

The system of claim 14, wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
receive environment data representing a three-dimensional map of the environment;
determine, using the environment data, a first direction in which the first image capture component is directed while the device is at the first position;
determine a first bounding box corresponding to a portion of the first image data representing the object; and
based at least in part on the first position data and the first direction, determine that the first bounding box corresponds to the second position data.

### Claim 21 (depends on 14)

The system of claim 14, wherein:
the first image capture component is attached to a distal end of the mechanical component,
the data representing the configuration of the mechanical component indicates at least one of a first angle and a first distance,
the first angle represents a direction in which the first image capture component is directed, and
the first distance indicates an amount that the first image capture component is extended from the device.