# US 2021/0241490 A1 · Image processing for tracking actions of individuals

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241490 A1
- Document ID: US20210241490A1
- Status: Published application
- Kind: A1
- Application number: 17/239,039
- Filed: 2021-04-23
- Published: 2021-08-05
- Earliest claimed: 2018-10-30
- Assignees: NCR Corporation
- Applicants: NCR Corporation
- Inventors: Brent Vance Zucker; Adam Justin Lieberman
- Web page: https://patentgrep.com/patent/US20210241490A1

## CPC

- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/74: {involving reference images or patches}
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30196: Human being; Person
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Cameras capture time-stamped images of predefined areas. Individuals and item are tracked in the images. A time-series set of images are processed to determine actions taken by the individuals with respect to the items or to determine relationships between the individuals to the items.

## Claims

### Claim 1

(canceled)

### Claim 2

A method comprising:
tracking a person and an item from a time-stamped stream of images;
assigning a person identifier for the person;
isolating item pixels from the images by removing first pixels associated with individuals and second pixels associated with known background objects;
providing an item identifier for the item based on the item pixels;
tracking movements of the item relative to the person from the images;
determining a relationship between the item and the person based on the tracking; and
assigning the item identifier to a virtual shopping cart maintained for the person identifier of the person when the relationship indicates that the person is in possession of the item.

### Claim 3 (depends on 2)

The method of claim 2 further comprising, removing the item identifier from the virtual shopping cart when a subsequent relationship determined from subsequent images indicates that the person is no longer in possession of the item.

### Claim 4 (depends on 2)

The method of claim 2 further comprising, processing a payment for the item and any other items assigned to the virtual shopping cart using a registered payment method associated with the person identifier when the person is tracked from the images leaving an an egress point.

### Claim 5 (depends on 2)

The method of claim 2 further comprising, providing item details for the item and any other items assigned to the virtual shopping cart to a transaction terminal when the person is tracked from the images as being present at the transaction terminal for processing a payment from the person for virtual cart items during a checkout.

### Claim 6 (depends on 2)

The method of claim 2 further comprising, providing item details for the item and any other items assigned to the virtual shopping cart to a mobile application of a mobile device operated by the person for processing a payment from the person for virtual cart items during a checkout.

### Claim 7 (depends on 2)

The method of claim 2, wherein tracking the person and the item further includes processing the images as sets of a time series of image frames that tracks the person within a store and with the item appearing in at least some of the image frames with the person.

### Claim 8 (depends on 7)

The method of claim 7, wherein tracking the movements further includes calculating from the image frames a velocity and a direction of the movements of the item toward or away from a shelf.

### Claim 9 (depends on 8)

The method of claim 8, wherein determining further includes assigning an in-possession relationship to the relationship when the velocity exceeds a threshold velocity and the direction is away from the shelf.

### Claim 10 (depends on 8)

The method of claim 8, wherein determining further includes assigning a dispossessed relationship to the relationship when the velocity exceeds a threshold velocity and the direction is toward the shelf.

### Claim 11 (depends on 8)

The method of claim 8, wherein determining further includes assigning a touched relationship to the relationship based on the velocity and the direction of the movement of the item.

### Claim 12 (depends on 2)

The method of claim 2, wherein determining further includes processing rules associated with a velocity and a direction of the movements for the item within the images and assigning the relationship as an in-possession relationship, a dispossessed relationship, or a touched relationship.

### Claim 13

A method comprising:
processing images by tracking a person represented within the images relative to items represented within the images;
isolating each identified item within the images to item pixels by removing individual pixels associated with individuals and removing background pixels associated with background objects;
assigning item identifiers to the items based on the corresponding item pixels associated with each identified item;
tracking movements of the items within the images relative to the person and relative to known shelf locations of the items;
adding first item identifiers associated with first items of the items to a virtual shopping cart maintained for a person identifier for the person based on relationships between the person and the first items determined from the tracking.

### Claim 14 (depends on 13)

The method of claim 13 further comprising, removing a select first item identifier from the virtual shopping cart when a given relationship for the select first item identifier changes from an in-possession relationship to a dispossessed relationship indicating the the person removed a select first item associated with the select first item identifier from a cart associated with the virtual shopping cart.

### Claim 15 (depends on 13)

The method of claim 13 further comprising, processing a payment for the first items using payment details associated with or provided by the person when the person is tracked from the images exiting an egress point, tracked as being present at a transaction terminal within the store, or tracked as requesting checkout via a mobile application of a mobile device operated by the person within the store.

### Claim 16 (depends on 13)

The method of claim 13, wherein tracking further includes calculating a velocity and direction of the movements for each item from the images, processing rules associated with the velocity and the direction, and determining each relationship between the corresponding item and the person based on the processing of the rules.

### Claim 17 (depends on 13)

The method of claim 13, wherein tracking further includes maintaining metadata for the person, the item identifiers, and the relationships as the images are sequentially processed as a time series.

### Claim 18 (depends on 17)

The method of claim 17, wherein maintaining further includes updating the metadata associated with the relationships as changes in the movements of the items relative to the person are detected.

### Claim 19 (depends on 18)

The method of claim 18, wherein adding further includes finalizing the relationships and the first item identifiers associated with the virtual shopping cart when the person is detected as taking an action associated an intention of the person to check out of a store with the first items and pay for the first items.

### Claim 20

A system comprising:
cameras situated throughout a store;
a server comprising a processor and a non-transitory computer-readable storage medium;
the non-transitory computer-readable storage medium comprising executable instructions;
the executable instructions executed by the processor from the non-transitory computer-readable storage medium causing the processor to perform operations comprising:
receiving time-stamped images from the cameras;
tracking persons via assigned person identifiers from the images;
tracking item identifiers for items from the time-stamped images that are associated with the persons by isolating item pixels for other pixels associated with persons and background objects of the store from the time-stamped images and matching the corresponding item pixels to the corresponding item identifiers for the corresponding items;
maintaining relationships between the item identifiers and the person identifiers based on movements of the items relative the persons detected from the tracking of the persons and the tracking of the items; and
assigning select items to virtual shopping carts associated with the person identifiers when the corresponding relationships indicate that the corresponding persons are in possession of the select items.

### Claim 21 (depends on 20)

The system of claim 20, wherein the executable instructions associated with maintaining the relationships further includes resolving the relationships by providing the time-stamped images, the person identifiers, and the item identifiers as input to the trained neural network and receiving as output from the trained neural network relationship identifiers.
