# US 11,113,541 B2 · Detection of object removal and replacement from a shelf

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,113,541 B2
- Document ID: US11113541B2
- Status: Granted
- Kind: B2
- Application number: 17/104,582
- Filed: 2020-11-25
- Published: 2021-04-29
- Granted: 2021-09-07
- Earliest claimed: 2019-10-25
- Assignees: 7-Eleven, Inc.
- Applicants: 7-Eleven, Inc.
- Inventors: Sumedh Vilas Datar; Sailesh Bharathwaaj Krishnamurthy; Shahmeer Ali Mirza
- Web page: https://patentgrep.com/patent/US11113541B2

## CPC

- H04N 7/183: {for receiving images from a single remote source}
- G06T 7/215: Motion-based segmentation
- G06T 7/292: Multi-camera tracking
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G08B 13/19608: {Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position}
- G08B 13/19613: {Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion}
- G08B 13/19645: {Multiple cameras, each having view on one of a plurality of scenes, e.g. multiple cameras for multi-room surveillance or for tracking an object by view hand-over}
- G08B 13/19669: {Event triggers storage or change of storage policy}
- G06T 2207/30208: Marker matrix
- G06V 20/44: {Event detection}
- G06V 20/68: Food, e.g. fruit or vegetables
- G06V 2201/07: Target detection
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS

## Abstract

An image sensor is positioned such that a field-of-view of the image sensor encompasses at least a portion of a rack storing items. The image sensor generates angled-view images of the items stored on the rack. A tracking subsystem determines that a person has interacted with the rack and receives image frames of the angled-view images. The tracking subsystem determines that the person interacted with a first item stored on the rack. A first image is identified associated with a first time before the person interacted with the first item, and a second image is identified associated with a second time after the person interacted with the first item. If it is determined, based on a comparison of the first and second images, that the item was removed from the rack, the first item is assigned to the person.

## Claims

### Claim 1

A system, comprising:
a rack comprising shelves configured to store items;
an image sensor positioned such that a field-of-view of the image sensor encompasses at least a portion of the rack, wherein the image sensor is configured to generate angled-view images of the items stored on the shelves of the rack; and
a tracking subsystem coupled to the image sensor, the tracking subsystem comprising at least one processor configured to:
determine that a person has interacted with the rack;
receive an image feed comprising frames of the angled-view images generated by the image sensor after the person has interacted with the rack;
determine, based on at least one angled-view image of the image feed, that the person interacted with a first item stored on the rack;
select a first image from the image feed associated with a first time before the person interacted with the first item, and a second image from the image feed associated with a second time after the person interacted with the first item;
over a period of time, track a pixel position of a wrist of the person in the image feed;
determine, based on the tracked pixel position of the wrist, a region-of interest defining a portion of the first image and a portion of the second image;
determine, based on a comparison of the portion of the first image defined by the region-of-interest to the portion of the second image defined by the region-of-interest, whether the first item was removed from the rack or the item was placed on the rack;
if it is determined that the first item was removed from the rack, assign the first item to the person; and
if it is determined that the item was placed on the rack, unassign the first item from the person.

### Claim 2 (depends on 1)

The system of claim 1, wherein the processor is further configured to determine that the person has interacted with the rack by determining that an arm of the person has passed within a zone adjacent to a front of the rack.

### Claim 3 (depends on 1)

The system of claim 1, wherein the processor is further configured to determine that the person has interacted with the rack by:
determining that the tracked pixel position of the wrist corresponds to a position on a shelf of the rack.

### Claim 4 (depends on 1)

The system of claim 1, wherein:
the system further comprises a weight sensor on which the first item is disposed on the shelf of the rack; and
the processor is further communicatively coupled to the weight sensor and configured to determine that the person interacted with the first item by receiving a signal from the weight sensor indicating a change in weight at the weight sensor on which the first item is disposed.

### Claim 5 (depends on 1)

The system of claim 1, wherein the processor is further configured to determine that the person interacted with the first item by:
determining, based on the tracked pixel positions, an aggregated wrist position corresponding to a maximum depth within the rack to which the wrist extends over the period of time; and
determining that the aggregated wrist position is within a threshold distance of a predefined position of the first item.

### Claim 6 (depends on 1)

The system of claim 1, wherein the processor is further configured to:
identify an interaction time associated with the person interacting with the shelf;
determine the first time as the interaction time minus a first predefined time interval;
identify the first image as an image from the image feed at or near the first time;
determine the second time as the interaction time plus a second predefined time interval; and
identify the second image as an image from the image feed at or near the second time.

### Claim 7 (depends on 1)

The system of claim 1, wherein the processor is further configured to determine whether the first item was removed from the rack or the item was placed on the rack by:
providing the portion of the first image and the portion of the second image to a machine learning model that is trained to determine a probability corresponding to whether an item has been added or removed based on a comparison of two input images;
if the probability determined by the neural network is greater than or equal to a threshold value, determine that the first item was added to the rack; and
if the probability determined by the neural network is less than the threshold value, determine that the first item was removed from the rack.

### Claim 8 (depends on 1)

The system of claim 1, wherein the processor is further configured to:
maintain a digital shopping cart for the person;
if it is determined that the item was removed from the rack, assign the item to the person by adding the first item to the digital shopping cart; and
if it is determined that the item was placed on the rack, unassign the item from the person by removing the first item from the digital shopping cart.

### Claim 9

A method, comprising:
determining that a person has interacted with a rack comprising shelves configured to store items;
receive an image feed comprising frames of angled-view images generated by an image sensor after the person has interacted with the rack, wherein the image sensor is positioned such that a field-of-view of the image sensor encompasses at least a portion of the rack, wherein the image sensor is configured to generate the angled-view images of the items stored on the shelves of the rack;
determining, based on at least one angled-view image of the image feed, that the person interacted with a first item stored on the rack;
selecting a first image from the image feed associated with a first time before the person interacted with the first item, and a second image from the image feed associated with a second time after the person interacted with the first item;
over a period of time, tracking a pixel position of a wrist of the person in the image feed;
determining, based on the tracked pixel position of the wrist, a region-of interest defining a portion of the first image and a portion of the second image;
determining, based on a comparison of the portion of the first image defined by the region-of-interest to the portion of the second image defined by the region-of-interest, whether the first item was removed from the rack or the item was placed on the rack;
if it is determined that the first item was removed from the rack, assigning the first item to the person; and
if it is determined that the first item was placed on the rack, unassign the first item from the person.

### Claim 10 (depends on 9)

The method of claim 9, further comprising determining that the person has interacted with the rack by determining that an arm of the person has passed within a zone adjacent to a front of the rack.

### Claim 11 (depends on 9)

The method of claim 9, further comprising determining that the person has interacted with the rack by:
determining that the tracked pixel position of the wrist corresponds to a position on a shelf of the rack.

### Claim 12 (depends on 9)

The method of claim 9, further comprising determining that the person interacted with the first item by receiving a signal from a weight sensor indicating a change in weight at the weight sensor on which the first item is disposed.

### Claim 13 (depends on 9)

The method of claim 9, further comprising determining that the person interacted with the first item by:
determining, based on the tracked pixel positions, an aggregated wrist position corresponding to a maximum depth within the rack to which the wrist extends over the period of time; and
determining that the aggregated wrist position is within as threshold distance of a predefined position of the first item.

### Claim 14 (depends on 9)

The method of claim 9, further comprising:
identifying an interaction time associated with the person interacting with the shelf;
determining the first time as the interaction time minus a first predefined time interval;
identifying the first image as an image from the image feed at or near the first time;
determining the second time as the interaction time plus a second predefined time interval; and
identifying the second image as an image from the image feed at or near the second time.

### Claim 15 (depends on 9)

The method of claim 9, further comprising determining whether the first item was removed from the rack or the item was placed on the rack by:
providing the portion of the first image and the portion of the second image to a machine learning model that is trained to determine a probability corresponding to whether an item has been added or removed based on a comparison of two input images;
if the probability determined by the neural network is greater than or equal to a threshold value, determine that the first item was added to the rack; and
if the probability determined by the neural network is less than the threshold value, determine that the first item was removed from the rack.

### Claim 16 (depends on 9)

The method of claim 9, further comprising:
maintaining a digital shopping cart for the person;
if it is determined that the item was removed from the rack, assigning the item to the person by adding the first item to the digital shopping cart; and
if it is determined that the item was placed on the rack, unassigning the item from the person by removing the first item from the digital shopping cart.

### Claim 17

A tracking subsystem comprising at least one processor configured to:
determine that a person has interacted with a rack comprising shelves configured to store items;
receive an image feed comprising frames of angled-view images generated by an image sensor after the person has interacted with the rack, wherein the image sensor is positioned such that a field-of-view of the image sensor encompasses at least a portion of the rack, wherein the image sensor is configured to generate the angled-view images of the items stored on the shelves of the rack;
determine, based on at least one angled-view image of the image feed, that the person interacted with a first item stored on the rack;
select a first image from the image feed associated with a first time before the person interacted with the first item, and a second image from the image feed associated with a second time after the person interacted with the first item;
over a period of time, track a pixel position of a wrist of the person in the image feed;
determine, based on the tracked pixel position of the wrist, a region-of interest defining a portion of the first image and a portion of the second image;
determine, based on a comparison of the portion of the first image defined by the region-of-interest to the portion of the second image defined by the region-of-interest, whether the first item was removed from the rack or the item was placed on the rack;
if it is determined that the first item was removed from the rack, assign the first item to the person; and
if it is determined that the first item was placed on the rack, unassign the first item from the person.

### Claim 18 (depends on 17)

The tracking subsystem of claim 17, wherein the processor is further configured to determine that the person has interacted with the rack by determining that an arm of the person has passed within a zone adjacent to a front of the rack.

### Claim 19 (depends on 17)

The tracking subsystem of claim 17, wherein the processor is further configured to determine that the person has interacted with the rack by:
determining that the tracked pixel position of the wrist corresponds to a position on a shelf of the rack.

### Claim 20 (depends on 17)

The tracking subsystem of claim 17, wherein the processor is further configured to determine that the person interacted with the first item by receiving a signal from a weight sensor indicating a change in weight at the weight sensor on which the first item is disposed.

### Claim 21 (depends on 17)

The tracking subsystem of claim 17, wherein the processor is further configured to determine that the person interacted with the first item by:
determining, based on the tracked pixel positions, an aggregated wrist position corresponding to a maximum depth within the rack to which the wrist extends over the period of time; and
determining that the aggregated wrist position is within as threshold distance of a predefined position of the first item.

### Claim 22 (depends on 17)

The tracking subsystem of claim 17, wherein the processor is further configured to:
identify an interaction time associated with the person interacting with the shelf;
determine the first time as the interaction time minus a first predefined time interval;
identify the first image as an image from the image feed at or near the first time;
determine the second time as the interaction time plus a second predefined time interval; and
identify the second image as an image from the image feed at or near the second time.

### Claim 23 (depends on 17)

The tracking subsystem of claim 17, wherein the processor is further configured to determine whether the first item was removed from the rack or the item was placed on the rack by:
providing the portion of the first image and the portion of the second image to a neural network trained to determine a probability corresponding to whether an item has been added or removed based on a comparison of two input images;
if the probability determined by the neural network is greater than or equal to a threshold value, determine that the first item was added to the rack; and
if the probability determined by the neural network is less than the threshold value, determine that the first item was removed from the rack.

### Claim 24 (depends on 17)

The tracking subsystem of claim 17, wherein the processor is further configured to:
maintain a digital shopping cart for the person;
if it is determined that the item was removed from the rack, assign the item to the person by adding the first item to the digital shopping cart; and
if it is determined that the item was placed on the rack, unassign the item from the person by removing the first item from the digital shopping cart.