# US 2021/0334541 A1 · Detecting shelf interactions using a sensor array

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334541 A1
- Document ID: US20210334541A1
- Status: Published application
- Kind: A1
- Application number: 17/368,204
- Filed: 2021-07-06
- Published: 2021-10-28
- Earliest claimed: 2019-10-25
- Assignees: 7-Eleven, Inc.
- Applicants: 7-Eleven, Inc.
- Inventors: Sailesh Bharathwaaj Krishnamurthy; Shahmeer Ali Mirza; Sarath Vakacharla; Trong Nghia Nguyen; Crystal Maung; Deepanjan Paul
- Web page: https://patentgrep.com/patent/US20210334541A1

## CPC

- G06T 7/248: {involving reference images or patches}
- G06T 7/215: Motion-based segmentation
- G06T 7/251: {involving models}
- G06T 7/292: Multi-camera tracking
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 10/764: using classification, e.g. of video objects
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06V 40/107: {Static hand or arm}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/30196: Human being; Person
- G06T 2207/30204: Marker
- G06T 2207/30232: Surveillance
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An object tracking system includes a sensor and a tracking system. The sensor is configured to capture a frame of at least a portion of a physical space within a global plane for a space. The tracking system is configured to receive the frame, to detect an object within a zone of the frame, and to determine a pixel location for the object. The tracking system is further configured to identify a zone of the physical structure based on the pixel location, to identify an item based on the identified zone.

## Claims

### Claim 1

An object tracking system, comprising:
a sensor configured to capture a frame of at least a portion of a physical structure within a global plane for a space, wherein:
the global plane represents (x,y) coordinates for the space;
the frame comprises a plurality of pixels;
each pixel from the plurality of pixels is associated with a pixel location comprising a pixel row and a pixel column;
the frame further comprises:
a first zone corresponding with a first portion of the physical structure in the global plane;
a second zone corresponding with a second portion of the physical structure in the global plane; and
a third zone proximate to the first zone and the second zone; and
the physical structure is partitioned by the first zone in the global plane and the second zone in the global plane;
a tracking system operably coupled to the sensor, comprising:
one or more memories configured to store:
an item map configured to associate:
a first item with the first zone of the physical structure; and
a second item with the second zone of the physical structure;
one or more processors operably coupled to the one or more memories, and configured to:
receive the frame;
detect an object within the third zone of the frame;
determine a pixel location for the object, wherein the pixel location comprises a first pixel row and a first pixel column;
identify one of the first zone and the second zone based on the first pixel row and the first pixel column of the pixel location for the object; and
identify an item in the item map based at least in part upon the identified zone.

### Claim 2 (depends on 1)

The system of claim 1, wherein:
the first zone is associated with a first range of pixels in the frame;
the second zone is associated with a second range of pixels in the frame; and
identifying one of the first zone and the second zone comprises:
identifying the first zone when the pixel location for the object is within the first range of pixels in the frame; and
identifying the second zone when the pixel location for the object is within the second range of pixels in the frame.

### Claim 3 (depends on 1)

The system of claim 1, wherein the memory is further configured to store a digital cart associated with a person and the one or more processors are further configured to add the identified item to the digital cart associated with the person.

### Claim 4 (depends on 3)

The system of claim 3, wherein:
the frame further comprises a fourth zone proximate to a front of the physical structure and the third zone; and
the one or more processors are further configured to:
determine a second pixel location for the person, wherein the second pixel location comprises a second pixel row and a second pixel column in the frame; and
determine the person is within the fourth zone before adding the identified item to the digital cart associated with the person.

### Claim 5 (depends on 2)

The system of claim 2, further comprising a weight sensor disposed in the identified zone of the physical structure, wherein the weight sensor is configured to measure a weight for items on the weight sensor; and
wherein the one or more processors are further configured to:
determine a weight decrease amount on the weight sensor;
determine an item quantity based on the weight decrease amount; and
add the identified item quantity to the digital cart associated with the person.

### Claim 6 (depends on 2)

The system of claim 2, wherein the one or more processors are further configured to:
determine a second pixel location for the person, wherein the second pixel location comprises a second pixel row and a second pixel column in the frame;
determine a third pixel location for a second person, wherein the third pixel location comprises a third pixel row and a third pixel column;
determine a first distance between the pixel location of the object and the second pixel location for the person;
determine a second distance between the pixel location of the object and the third pixel location for the second person; and
determine the first distance is less than the second distance before adding the identified item to the digital cart associated with the person.

### Claim 7 (depends on 1)

The system of claim 1, wherein:
the sensor is one of a plurality of sensors configured as a sensor array; and
the sensor array is positioned parallel with the global plane.

### Claim 8 (depends on 1)

The system of claim 1, wherein:
the one or more memories are further configured to store a homography associated with the sensor; and
the homography is configured to translate between pixel locations in the frame and (x,y) coordinates in the global plane.

### Claim 9 (depends on 2)

The system of claim 2, wherein:
the one or more memories are further configured to store a homography associated with the sensor, the homography configured to translate between pixel locations in the frame and (x,y) coordinate in the global plane; and
the one or more processors are further configured to:
determine a second pixel location for the person, wherein the second pixel location comprises a second pixel row and a second pixel column in the frame; and
apply the homography to the second pixel location to determine a first (x,y) coordinate in the global plane.

### Claim 10 (depends on 9)

The system of claim 9, wherein the one or more processors are further configured to identify the rack based on the first (x,y) coordinate.

### Claim 11

An object tracking method, comprising:
receiving from a sensor a frame of at least a portion of a physical structure within a global plane for a space, wherein:
the global plane represents (x,y) coordinates for the space;
the frame comprises a plurality of pixels;
each pixel from the plurality of pixels is associated with a pixel location comprising a pixel row and a pixel column;
the frame further comprises:
a first zone corresponding with a first portion of the physical structure in the global plane; and
a second zone corresponding with a second portion of the physical structure in the global plane; and
a third zone proximate to the first zone and the second zone;
storing an item map configured to associate:
a first item with the first zone of the physical structure;
a second item with the second zone of the physical structure;
detecting an object within a third zone of the frame;
determining a pixel location for the object, wherein the pixel location comprises a first pixel row and a first pixel column;
identifying one of the first zone and the second zone based on the first pixel row and the first pixel column of the pixel location for the object;
identifying an item in the item map based at last in part upon the identified zone.

### Claim 12 (depends on 11)

The method of claim 11, further comprising:
storing a digital cart associated with a person; and
adding the identified item to the digital cart associated with the person.

### Claim 13 (depends on 12)

The method of claim 12, further comprising:
determining a second pixel location for the person, wherein the second pixel location comprises a second pixel row and a second pixel column in the frame; and
determining the person is within a fourth zone proximate to a front of the physical structure and the third zone before adding the identified item to the digital cart associated with the person.

### Claim 14 (depends on 12)

The method of claim 12, further comprising:
determining a weight decrease amount on a weight sensor;
determining an item quantity based on the weight decrease amount; and
adding the identified item quantity to the digital cart associated with the person.

### Claim 15 (depends on 12)

The method of claim 12, further comprising:
determining a second pixel location for the person, wherein the second pixel location comprises a second pixel row and a second pixel column in the frame;
determining a third pixel location for a second person, wherein the third pixel location comprises a third pixel row and a third pixel column;
determining a first distance between the pixel location of the object and the second pixel location for the person;
determining a second distance between the pixel location of the object and the third pixel location for the second person; and
determining the first distance is less than the second distance before adding the identified item to the digital cart associated with the person.

### Claim 16 (depends on 11)

The method of claim 11, further comprising:
determining a second pixel location for the person, wherein the second pixel location comprises a second pixel row and a second pixel column in the frame; and
applying a homography to the second pixel location to determine a first (x,y) coordinate in the global plane, wherein the homography is configured to translate between pixel locations in the frame and (x,y) coordinates in the global plane.

### Claim 17

An object tracking system, comprising:
a sensor configured to capture a frame of at least a portion of a physical structure within a global plane for a space, wherein:
the global plane represents (x,y) coordinates for the space;
the frame comprises a plurality of pixels;
each pixel from the plurality of pixels is associated with a pixel location comprising a pixel row and a pixel column;
and
the frame further comprises a first zone proximate to a front of the rack; and
a tracking system operably coupled to the sensor, comprising:
one or more memories configured to store a digital cart associated with a person; and
one or more processors operably coupled to the one or more memories, configured to:
receive the frame;
detect an object within the first zone of the frame;
determine a first pixel location for the object, wherein the first pixel location comprises a first pixel row and a first pixel column;
determine a second pixel location for the person, wherein the second pixel location comprises a second pixel row and a second pixel column in the frame;
apply a homography to the second pixel location to determine a first (x,y) coordinate in the global plane, wherein the homography is configured to translate between pixel locations in the frame and (x,y) coordinates in the global plane;
identify a portion of the physical structure based on the first pixel row and the first pixel column of the pixel location for the object;
identify an item based at last in part upon the identified portion of the physical structure; and
add the identified item to the digital cart associated with the person.

### Claim 18 (depends on 17)

The system of claim 17, wherein:
the frame further comprises a second zone proximate to the front of the physical structure and the first zone; and
the one or more processors are further configured to:
determine a second pixel location for the person, wherein the second pixel location comprises a second pixel row and a second pixel column in the frame; and
determine the person is within the second zone before adding the identified item to the digital cart associated with the person.

### Claim 19 (depends on 17)

The system of claim 17, further comprising a weight sensor disposed on the physical structure, and
wherein the one or more processors are further configured to:
determine a weight decrease amount on the weight sensor;
determine an item quantity based on the weight decrease amount; and
add the identified item quantity to the digital cart associated with the person.

### Claim 20 (depends on 17)

The system of claim 17, wherein the one or more processors are further configured to:
determine a second pixel location for the person, wherein the second pixel location comprises a second pixel row and a second pixel column in the frame;
determine a third pixel location for a second person, wherein the third pixel location comprises a third pixel row and a third pixel column;
determine a first distance between the pixel location of the object and the second pixel location for the person;
determine a second distance between the pixel location of the object and the third pixel location for the second person; and
determine the first distance is less than the second distance before adding the identified item to the digital cart associated with the person.

### Claim 21 (depends on 17)

The system of claim 17, wherein:
the sensor is one of a plurality of sensors configured as a sensor array; and
the sensor array is positioned parallel with the global plane.
