# US 2021/0342903 A1 · Distributed and Automated Transaction Systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342903 A1
- Document ID: US20210342903A1
- Status: Published application
- Kind: A1
- Application number: 17/378,165
- Filed: 2021-07-16
- Published: 2021-11-04
- Earliest claimed: 2016-12-09
- Assignees: 365 Retail Markets, LLC
- Applicants: 365 Retail Markets, LLC
- Inventors: Paul McDonald; Ashwath Rajan; Tyler Garvin
- Web page: https://patentgrep.com/patent/US20210342903A1

## CPC

- G06Q 30/06: Buying, selling or leasing transactions
- G06F 18/2413: based on distances to training or reference patterns
- G06Q 10/087: Inventory or stock management, e.g. order filling, procurement or balancing against orders
- G06Q 10/08726: {for replenishment processing, procedures, or recommendations using forecasting or optimisation}
- G06Q 10/08772: {Inventory counting methods}
- G06Q 20/102: {Bill distribution or payments}
- G06Q 20/14: specially adapted for billing systems
- G06Q 20/20: Point-of-sale [POS] network systems
- G06Q 20/208: {Input by product or record sensing, e.g. weighing or scanner processing}
- G06Q 30/0201: Market modelling; Market analysis; Collecting market data
- G06Q 30/0202: Market predictions or forecasting for commercial activities
- G06Q 30/0207: Discounts or incentives, e.g. coupons or rebates
- G06Q 30/0241: Advertisements
- G06T 7/74: {involving reference images or patches}
- G06V 10/22: by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/95: {structured as a network, e.g. client-server architectures}
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 40/12: Fingerprints or palmprints
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G07G 1/0036: {Checkout procedures}
- G07G 1/0054: {with control of supplementary check-parameters, e.g. weight or number of articles}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30196: Human being; Person
- G06V 20/44: {Event detection}
- G06V 30/2247: {Characters composed of bars, e.g. CMC-7}
- G06V 40/172: {Classification, e.g. identification}
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G07G: REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS

## Abstract

Methods and apparatus are described that use machine vision and machine learning to eliminate many of the steps of a typical retail transaction. For example, implementations are contemplated in which a user simply removes an item from a retail location to effect the transaction.

## Claims

### Claim 1

A system, comprising:
a product enclosure including one or more shelves for placement of a plurality of products, one or more cameras, one or more processors, and a network connection, the one or more processors being configured to:
using the one or more cameras, capture a sequence of images associated with a transaction involving one or more of the products; and
using the network connection, transmit the sequence of images to a remote back end infrastructure; and
the back end infrastructure including one or more computing devices operating in a network, the one or more computing devices being configured to:
receive the sequence of images from the product enclosure via the network;
process the sequence of images to (1) identify a hand reaching into the product enclosure, (2) determine a location of the hand within the enclosure, and (3) identify a first product grasped by the hand; and
charge an account associated with the user a transaction amount for the transaction, the transaction amount including a first amount corresponding to the first product.

### Claim 2 (depends on 1)

The system of claim 1, wherein the one or more computing devices or the one or more processors are further configured to identify the user by one of (1) biometric identification, (2) detection of a machine readable code corresponding to the user, or (3) receiving a code corresponding to the product enclosure transmitted from a device corresponding to the user.

### Claim 3 (depends on 1)

The system of claim 1, wherein the one or more processors are further configured to, using the one or more cameras, determine that a human is in front of the product enclosure.

### Claim 4 (depends on 1)

The system of claim 1, wherein the one or more computing devices are further configured to process the sequence of images to discard one or more subsets of the images that do not include the hand.

### Claim 5 (depends on 1)

The system of claim 1, wherein the one or more computing devices are configured to process the sequence of images to determine a location of the hand within the enclosure based on a model of a spatial context of the product enclosure.

### Claim 6 (depends on 5)

The system of claim 5, wherein the one or more computing devices are configured to process the sequence of images to identify the first product grasped by the hand based on a product location schema associated with the spatial context.

### Claim 7 (depends on 1)

The system of claim 1, wherein the one or more computing devices are configured to process the sequence of images using a localization model to focus on portions of individual images in which the hand appears.

### Claim 8 (depends on 1)

The system of claim 1, wherein the one or more computing devices are further configured to process the sequence of images to identify one or more changes within the product enclosure between successive images.

### Claim 9 (depends on 1)

The system of claim 1, wherein the product enclosure further comprises one or more doors, and wherein the one or more processors are further configured to:
unlock the one or more doors in response to a message from the back end infrastructure; and
transmit the sequence of images to the back end infrastructure in response to the one or more doors locking, or in response to passage of a predetermined period of time.

### Claim 10 (depends on 1)

The system of claim 1, wherein the product enclosure further comprises one or more doors, and wherein the one or more processors are further configured to:
determine that the one or more doors have been open longer than a predetermined period of time;
using the one or more cameras, passively record video; and
using the network connection, periodically transmit a frame of the video to the back end infrastructure.

### Claim 11 (depends on 10)

The system of claim 10, wherein the one or more computing devices are further configured to:
process the frames of the video received from the product enclosure to identify one or more changes within the product enclosure between successive ones of the frames; and
request upload of the video from the product enclosure in response to identification of the one or more changes.

### Claim 12

A product enclosure, comprising:
one or more shelves for placement of a plurality of products;
one or more cameras;
a network connection; and
one or more processors configured to:
using the one or more cameras, capture a sequence of images associated with a transaction involving one or more of the products; and
using the network connection, transmit the sequence of images to a remote back end infrastructure.

### Claim 13 (depends on 12)

The product enclosure of claim 12, wherein the one or more processors are further configured to facilitate identification of the user by one of (1) capturing biometric identifying information, (2) detecting a machine readable code corresponding to the user, or (3) receiving a code transmitted from a device corresponding to the user.

### Claim 14 (depends on 12)

The product enclosure of claim 12, wherein the one or more processors are further configured to, using the one or more cameras, determine that a human is in front of the product enclosure.

### Claim 15 (depends on 12)

The product enclosure of claim 12, further comprising one or more doors, and wherein the one or more processors are further configured to:
unlock the one or more doors in response to a message from a back end infrastructure; and
transmit the sequence of images to the back end infrastructure in response to the one or more doors locking, or in response to passage of a predetermined period of time.

### Claim 16 (depends on 12)

The product enclosure of claim 12, further comprising one or more doors, and wherein the one or more processors are further configured to:
determine that the one or more doors have been open longer than a predetermined period of time;
using the one or more cameras, passively record video; and
using the network connection, periodically transmit a frame of the video to a back end infrastructure.

### Claim 17

A system, comprising one or more computing devices operating in a network, the one or more computing devices being configured to:
receive a sequence of images representing a transaction from a product enclosure via the network;
process the sequence of images to (1) identify a hand reaching into the product enclosure, (2) determine a location of the hand within the enclosure, and (3) identify a first product grasped by the hand; and
charge an account associated with the user a transaction amount for the transaction, the transaction amount including a first amount corresponding to the first product.

### Claim 18 (depends on 17)

The system of claim 17, wherein the one or more computing devices are further configured to identify the user by one of (1) biometric identification, (2) detection of a machine readable code corresponding to the user, or (3) receiving a code corresponding to the product enclosure transmitted from a device corresponding to the user.

### Claim 19 (depends on 17)

The system of claim 17, wherein the one or more computing devices are further configured to process the sequence of images to discard one or more subsets of the images that do not include the hand.

### Claim 20 (depends on 17)

The system of claim 17, wherein the one or more computing devices are configured to process the sequence of images to determine a location of the hand within the enclosure based on a model of a spatial context of the product enclosure.
