US 12,592,096 B1Granted

Modeling and detecting shopping events using visual images and machine learning

AssigneeAmazon Technologies, Inc.

InventorsMichael Dillon, Gerard Guy Medioni, Ali Rahimi

Life of this patent

Estimating the term…

Loading patent…

Abstract

Cameras installed at a facility capture images over periods of time and detect features such as locations of hands, heads or other body parts within the images. Time series of data regarding such features are provided to machine learning systems to determine whether an event such as a taking or a return of the item is depicted within the images, or whether no such event is detected. The time series are provided to the machine learning system in an iterative manner, with time series associated with a set of baseline features provided to the machine learning system first. If an event is not detected and associated with an actor based on the baseline features, time series associated with supplemental features are successively provided to the machine learning system, along with the time series associated with the baseline features, until the event is detected and associated with an actor.

Claims20

Dependent claims

A method comprising:

capturing at least a first plurality of images by a first imaging device having a first field of view, wherein the first plurality of images are captured over a first period of time;
detecting, by the first imaging device, locations of at least a first body part of a first actor depicted within at least a subset of the first plurality of images;
detecting, by the first imaging device, locations of at least a second body part of a second actor depicted within the subset of the first plurality of images;
determining, by the first imaging device, first values of data corresponding to at least a first feature of the first actor based at least in part on at least one of the locations of at least the first body part of the first actor depicted within at least the subset of the first plurality of images;
determining, by the first imaging device, second values of data corresponding to at least a second feature of the second actor based at least in part on at least one of the locations of at least the second body part of the second actor depicted within at least the subset of the first plurality of images;
constructing, by the first imaging device, a first time series of data, wherein the first time series of data comprises the first values of data corresponding to the first feature and statuses of the first body part of the first actor at times at which each of the subset of the first plurality of images was captured;
constructing, by the first imaging device, a second time series of data, wherein the second time series of data comprises the second values of data corresponding to the second feature and statuses of the second body part of the second actor at the times at which each of the subset of the first plurality of images was captured;
providing at least a portion of the first time series of data as a first input to a first machine learning system, wherein the first machine learning system is trained to determine whether at least one actor is associated with at least one of a plurality of events based at least in part on a feature of the at least one actor;
receiving at least a first output from the first machine learning system in response to the first input;
providing at least a portion of the second time series of data as a second input to the first machine learning system;
receiving at least a second output from the first machine learning system in response to the second input;
and
determining, based at least in part on the first output and the second output, that the first actor is associated with a first event of the plurality of events,
wherein the first event comprises one of:
placing at least one item on a storage unit within the first field of view;
or
removing at least one item from the storage unit.

14 more elements

A system comprising:

a camera having a field of view including at least a portion of at least one surface for accommodating one or more items, wherein the camera comprises a processor unit and an optical sensor;
and
a computer system in communication with the camera, wherein the computer system is configured to execute a machine learning system trained to determine whether at least one actor is associated with at least one of a plurality of events based at least in part on a feature of the at least one actor detected in at least one image,
wherein the processor unit is programmed with one or more sets of instructions that, when executed by the processor unit, cause the camera to execute a first method comprising:
capturing a plurality of images over a period of time;
detecting locations of at least a first body part of a first actor depicted within at least a subset of the plurality of images;
detecting locations of at least a second body part of a second actor depicted within the subset of the plurality of images;
determining first values of data corresponding to at least a first feature of the first actor based at least in part on at least one of the locations of at least the first body part of the first actor depicted within at least the subset of the plurality of images;
determining second values of data corresponding to at least a second feature of the second actor based at least in part on at least one of the locations of at least the second body part of the second actor depicted within at least the subset of the first plurality of images;
constructing, by the first imaging device, a first time series of data, wherein the first time series of data comprises the first values of data corresponding to the first feature and statuses of the first body part of the first actor at times at which each of the subset of the plurality of images was captured;
constructing, by the first imaging device, a second time series of data, wherein the second time series of data comprises the second values of data corresponding to the second feature and statuses of the second body part of the second actor at the times at which each of the subset of the plurality of images was captured;
and
transmitting at least the first time series of data and the second time series of data to the computer system,
wherein the computer system is programmed with one or more sets of instructions that, when executed by the processor unit, cause the computer system to execute a second method comprising:
providing at least a portion of the first time series of data as a first input to the machine learning system;
receiving at least a first output from the machine learning system in response to the first input;
providing at least a portion of the second time series of data as a second input to the machine learning system;
receiving at least a second output from the machine learning system in response to the second input;
and
determining, based at least in part on the first output and the second output, that the first actor is associated with one of the plurality of events,
wherein the one of the plurality of events comprises one of:
placing at least one item on the at least one surface;
or
removing at least one item from the at least one surface.

21 more elements

A camera having a field of view including at least a portion of at least one surface for accommodating one or more items, wherein the camera comprises a processor unit and an optical sensor,

wherein the processor unit is configured to execute a machine learning system trained to determine whether at least one actor is associated with at least one of a plurality of events based at least in part on a feature of the at least one actor detected in at least one image,
wherein the processor unit is programmed with one or more sets of instructions that, when executed by the processor unit, cause the camera to execute a method comprising:
capturing a plurality of images over a period of time;
detecting locations of at least a first body part of a first actor depicted within at least a subset of the plurality of images;
detecting locations of at least a second body part of a second actor depicted within the subset of the plurality of images;
determining first values of data corresponding to at least a first feature of the first actor based at least in part on at least one of the locations of at least the first body part of the first actor depicted within at least the subset of the plurality of images;
determining second values of data corresponding to at least a second feature of the second actor based at least in part on at least one of the locations of at least the second body part of the second actor depicted within at least the subset of the plurality of images;
constructing, by the first imaging device, a first time series of data, wherein the first time series of data comprises the first values of data corresponding to the first feature and statuses of the first body part of the first actor at times at which each of the subset of the plurality of images was captured;
constructing, by the first imaging device, a second time series of data, wherein the second time series of data comprises the second values of data corresponding to the second feature and statuses of the second body part of the second actor at the times at which each of the subset of the first plurality of images was captured;
providing at least a portion of the first time series of data as a first input to the machine learning system;
receiving at least a first output from the machine learning system in response to the first input;
providing at least a portion of the second time series of data as a second input to the machine learning system;
receiving at least a second output from the machine learning system in response to the second input;
and
determining, based at least in part on the first output and the second output, that the first actor is associated with one of the plurality of events,
wherein the one of the plurality of events comprises one of:
placing at least one item on the at least one surface;
or
removing at least one item from the at least one surface.

16 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/038,490Filed 30 Sep 2020
Granted: 31 Mar 2026
Term ends: Not estimatedThe source records don’t support an estimate.

Classification10 codes

G06V 40/107
Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands: static hand or arm
G06F 18/28
Analysing: determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
G06V 10/774
Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or…:…
G06V 10/82
Using pattern recognition or machine learning: using neural networks