US 11,238,401 B1Granted

Identifying user-item interactions in an automated facility

AssigneeAmazon Technologies, Inc.

InventorsDanny Guan, Prathiban Mohanasundaram, Shannon Thomas, Mirko Mandic, Helene Causse, Brett Richard Taylor, Waqas Syed Ahmed, David Echevarria Ignacio

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

Techniques for employing user interfaces to output information indicative of events occurring in an inventory facility, and receive feedback from a human regarding the events are described herein. In one implementation, an event may take place in an inventory facility, such as a customer taking an item from an inventory location, returning an item to an inventory location, and so forth. An automated system of an inventory management system may process sensor data collected by sensors in the inventory facility to determine details of the event. In some examples, the inventory management system is unable to determine with a high level of confidence what occurred during the event. The inventory management system may provide the sensor data to a human associate through an associate interface, and receive input regarding details of the event from the human associate through the associate interface.

Claims20

Dependent claims

A method comprising:

receiving video data at a client computing device, the video data corresponding to a location of an environment for a period of time and depicting:
a first user at the location for at least a portion of the period of time;
a second user at the location for at least a portion of the period of time;
and
an item at the location for at least a portion of the period of time;
receiving first data representing first associate input at the client computing device, the first data indicating the video data depicts the first user performing a take of the item;
receiving second data representing second associate input at the client computing device, the second data indicating the video data depicts the second user performing the take of the item;
sending data representing a user interface (UI) to a display associated with the client computing device, the UI comprising:
a first portion in which the video data is presented;
and
a second portion, adjacent the first portion, in which data representing the first data and second data is presented, the second portion including a compare-answers control and a submit-answer control;
receiving input indicating a selection of the compare-answers control;
sending, at least partly in response to receiving the selection of the compare-answers control, data to the display to be presented in the second portion of the UI, the data comprising:
first graphical indicators representing the first data, the first graphical indicators including a first user image depicting the first user, an item image depicting the item, and text indicating the take of the item;
second graphical indicators representing the second data, the second graphical indicators including a second user image depicting the second user, the item image depicting the item, and the text indicating the take of the item;
a disregard control associated with the second user image depicting the second user;
and
a review-answer control associated with the second user image depicting the second user;
receiving input indicating a selection of the disregard control;
receiving input indicating a selection of the submit-answer control;
storing an indication of the first user performing the take of the item;
and
sending, to one or more server computing devices and at least partly in response to the receiving the input indicating the selection of the submit-answer control, data representing the indication of the first user performing the take of the item.

20 more elements

A computing device comprising:

one or more processors;
one or more input interfaces;
a display;
one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising:
receiving video data corresponding to a location of an environment for a period of time, the video data depicting a first user, a second user, and an item;
sending the video data to the display;
receiving first data representing first associate input, the first associate input indicating that the video data depicts the first user performing a first type of interaction with the item;
receiving second data representing second associate input, the second associate input indicating that the video data depicts the second user performing the first type of interaction with the item;
sending data representing text to the display, the text corresponding to a request to identify whether the first user or the second user interacted with the item depicted in the video data;
and
receiving first escalator associate input identifying the first user as having interacted with the item in the video data.

8 more elements

A method comprising:

receiving video data at a client computing device, video data corresponding to a location of an environment for a period of time and depicting:
a user at the location for at least a portion of the period of time;
a first item at the location for at least a portion of the period of time;
and
a second item at the location for at least a portion of the period of time;
sending the video data to a display associated with the client computing device;
receiving first data representing first associate input, the first associate input indicating that the video data depicts the user performing a first type of interaction with the first item;
receiving second data representing second associate input, the second associate input indicating that the video data depicts the user performing the first type of interaction with the second item;
sending data representing text to the display, the text corresponding to a request to identify whether the first item or the second item was interacted with by the user;
and
receiving first escalator associate input identifying the first item as having been interacted with by the user.

8 more elements

Dependencies are inferred from the claim text.

Facts

Application: 15/470,207Filed 27 Mar 2017
Granted: 1 Feb 2022
Term ends: Not estimatedThe source records don’t support an estimate.

Classification11 codes

G06Q 10/087
Logistics, e.g. warehousing, loading or distribution; Inventory or stock management: inventory or stock management, e.g. order filling,…
G06F 3/0481
Interaction techniques based on graphical user interfaces [GUI]: based on specific properties of the displayed interaction object or a…
G06F 3/04817
Based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like…:…
G06F 3/0484
Interaction techniques based on graphical user interfaces [GUI]: for the control of specific functions or operations, e.g. selecting or…