US 12,524,503 B1Granted

System and method for vision-based event detection

AssigneeAmazon Technologies, Inc.

InventorsJayakrishnan Kumar Eledath, Nikhil Chacko, Alessandro Bergamo, Kaustav Kundu, Marian Nasr Amin George, Jingjing Liu, Nishitkumar Ashokkumar Desai, Pahal Kamlesh Dalal, Keshav Nand Tripathi

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

This disclosure describes systems and techniques for identifying events that occur within an environment using image data captured at the environment. For example, one or more cameras may generate image data representative of a user interacting with an item on the shelf. This image data may be used to generate feature data associated with the user and the item, which may be analyzed by one or more classifiers for identifying an interaction between the user and the item. The systems and techniques may then generate interaction data, which in turn may be analyzed by one or more additional classifiers for identifying an event, such as the user picking a particular item from the shelf within the environment. Event data indicative of the event may then be used to update a virtual cart of the user.

Claims20

Dependent claims

A computer device comprising:

one or more processors;
memory;
a first component stored in the memory and executable on the one or more processors, wherein the first component is configured to:
receive image data from one or more cameras in an environment, wherein the image data represents a first user interacting with a first item in the environment;
and
generate, using the image data, feature data associated with at least one of the first user or the first item, wherein the feature data comprises at least one segmentation map representing a plurality of regions of pixels of the image data and labels for each of the plurality of regions of pixels, and wherein at least one of the labels corresponds to a portion of the first user;
a second component stored in the memory and executable on the one or more processors, wherein the second component is configured to:
receive the feature data;
input the feature data into a first artificial neural network that has been trained to determine whether the feature data represents the interaction between the first user and the first item;
and
generate, using the first artificial neural network, first data indicating at least a first time range of the interaction and a first location of the interaction;
a third component stored in the memory and executable on the one or more processors, wherein the third component is configured to:
receive the first data;
input the first data into a second artificial neural network that has been trained to determine whether data indicating a time range and a location represents at least one predefined activity of a plurality of predefined activities;
determine, using the second artificial neural network, that the first data represents a first predefined activity of the plurality of predefined activities;
receive second data associated with a virtual cart of the first user;
determine, based at least in part on the first data and the second data, a first action taken by the first user with respect to the first item and a first identity of the first item;
and
generate, based at least in part on the first data and the second data, third data indicating at least a second time range of the first predefined activity, a second location of the first predefined activity, the first action taken by the first user with respect to the first item, and the first identity of the first item;
a fourth component stored in the memory and executable on the one or more processors, wherein the fourth component is configured to:
receive the third data;
and
automatically update the virtual cart based at least in part on the third data to indicate the first identity of the first item and the first action taken with respect to the first item.

20 more elements

A method comprising:

generating, using one or more cameras in an environment that includes inventory locations holding items, image data representing a first user in the environment;
generating feature data based at least in part on the image data, wherein the feature data comprises a segmentation map, and wherein the segmentation map represents a plurality of sets of pixels of the image data and labels for each of the plurality of sets of pixels;
inputting the feature data into a first artificial neural network that has been trained to determine whether feature data represents an interaction between a user and an item;
generating, using the first artificial neural network based at least in part on the feature data, first interaction data indicative of a first interaction between the first user and a first item, wherein the interaction data indicates a time range and at least one inventory location of the environment;
inputting the first interaction data into a second artificial neural network that has been trained to determine whether the interaction data corresponds to one of a plurality of predefined activities in the environment;
determining, using the second artificial neural network based at least in part on the interaction data, that the first interaction data corresponds to a first predefined activity in the environment, wherein the first predefined activity comprises a first action performed by the first user with respect to the first item at the at least one inventory location;
receiving virtual cart data representing a current state of a virtual cart of the first user;
generating, based at least in part on the interaction data and the virtual cart data, first event data indicating the first action performed by the first user and a first identity of the first item;
and
automatically updating the virtual cart of the first user to indicate the first action performed by the first user and the first identity of the first item based at least in part on the first event data.

7 more elements

A system comprising:

one or more cameras, wherein each of the one or more cameras is configured to capture image data from an environment that includes inventory locations;
one or more processors;
and
one or more computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform operations comprising:
capturing, by the one or more cameras, image data from the environment;
generating feature data based at least in part on the image data, wherein the feature data comprises a segmentation map representing a plurality of sets of pixels of the image data and labels for each of the plurality of sets of pixels;
inputting the feature data into a first artificial neural network that has been trained to determine whether image data from which feature data was generated represents an interaction between a user and an item;
generating, using the first artificial neural network based at least in part on the feature data, first interaction data indicative of a first interaction between a first user and a first item, the first interaction data indicating a time range of the image data and at least one inventory location of the environment;
inputting the first interaction data into a second artificial neural network that has been trained to determine whether an interaction represented in interaction data corresponds to one of a plurality of predefined activities in the environment;
determining, using the second artificial neural network based at least in part on the first interaction data, that the first interaction data corresponds to at least a first predefined activity in the environment, wherein the first predefined activity comprises the first user performing a first action in the environment with respect to a first item at the at least one inventory location;
receiving virtual cart data representing a state of a first virtual cart of the first user;
generating, based at least in part on the first interaction data and the virtual cart data, first event data indicating the first action performed by the first user and an identity of the first item;
and
automatically updating the first virtual cart of the first user to indicate the first action performed by the first user and the identity of the first item based at least in part on the first data.

11 more elements

Dependencies are inferred from the claim text.

Facts

Application: 18/525,398Filed 30 Nov 2023 · continuation
Priority: 11 Feb 2019
Granted: 13 Jan 2026
Term ends: Not estimatedThe source records don’t support an estimate.

Classification13 codes

G06Q 30/0633
Electronic shopping [e-shopping]: managing shopping lists, e.g. compiling or processing purchase lists
G06F 18/241
Classification techniques: relating to the classification model, e.g. parametric or non-parametric approaches
G06Q 30/06334
Virtual shopping carts: comprising goods or services offered by multiple providers, e.g. multiple merchants
G06Q 30/0639
Electronic shopping [e-shopping]: locating goods or services, e.g. based on physical position of the goods or services within a shopping facility