# US 11,443,291 B2 · Tracking product items in an automated-checkout store

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,443,291 B2
- Document ID: US11443291B2
- Status: Granted
- Kind: B2
- Application number: 16/374,692
- Filed: 2019-04-03
- Published: 2020-06-11
- Granted: 2022-09-13
- Earliest claimed: 2018-12-05
- Assignees: AiFi Corp
- Applicants: AiFi Inc.
- Inventors: Steve Gu; Joao Falcao; Ying Zheng; Shuang Liu; Brian Bates; Juan Ramon Terven-Salinas; Carlos Ruiz; Tyler Crain
- Web page: https://patentgrep.com/patent/US11443291B2

## CPC

- G06Q 20/203: {Inventory monitoring}
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/00: Machine learning
- G06Q 20/201: {Price look-up processing, e.g. updating}
- G06Q 20/208: {Input by product or record sensing, e.g. weighing or scanner processing}
- G07G 1/0054: {with control of supplementary check-parameters, e.g. weight or number of articles}
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G07G: REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS

## Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for tracking a product item in an automated-checkout store. One of the methods includes receiving, from a plurality of sensors, data collected by the sensors, identifying an account associated with a person who enters the store, receiving data on movement of the product item collected by one or more of the sensors, determining an interaction between the person and the product item based on the data collected by one or more of the sensors, storing information associating the product item with the person; and deducting a payment amount from the identified account, wherein the payment amount is based on a price of the product item associated with the person.

## Claims

### Claim 1

A method for tracking a product item in an automated-checkout store, comprising:
receiving, by a computer system from a plurality of sensors including at least one or more cameras, one or more weight sensors, one or more pressure sensors, and one or more capacitive sensors, data collected by the sensors;
identifying, by the computer system, an account associated with a person who enters the store;
receiving, by the computer system, data on movement of the product item collected by the plurality of the sensors;
determining, by the computer system, an interaction between the person and the product item based on the data collected by the plurality of the sensors;
extracting, by the computer system, from the data received from the plurality of the sensors, a plurality of features associated with the product item, wherein the features comprise a conductivity, a color, a position, and a surface force image, the conductivity is used to determine a type of material that makes up at least a part of the product, and the surface force image is used to determine a footprint of the product that includes a shape of a bottom surface of the product;
determining, by the computer system, using a machine-learning model, an identity of the product item based on the extracted features including the conductivity, the color, the position, and the surface force image, wherein the machine-learning model is trained based on training data collected from the plurality of the sensors;
storing, by the computer system, information associating the product item with the person; and
deducting, by the computer system, a payment amount from the identified account, wherein the payment amount is based on a price of the product item associated with the person.

### Claim 2 (depends on 1)

The method of claim 1, wherein one or more of the sensors are affixed to one or more structures in the automated-checkout store, the one or more structures comprising any of:
a ceiling;
a floor;
a shelf;
a rack; and
a refrigerator.

### Claim 3 (depends on 1)

The method of claim 1, wherein the plurality of sensors further comprise one or more of:
a force sensor;
a vibration sensor;
a proximity sensor; and
a resistance-based film sensor.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
determining, by the computer system based on the data received from a first sensor of the plurality of the sensors, a first identity for the product item associated with a first confidence score;
determining, by the computer system based on the data received from a second sensor of the plurality of the sensors, a second identity for the product item associated with a second confidence score; and
selecting one of the first identity and the second identity as the identity of the product item based on a comparison of the first confidence score and the second confidence score.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
obtaining, by the computer system based on the data received from the plurality of the sensors, a plurality of values associated with a characteristic of the product item, wherein each of the values is determined based on the data received from one of the plurality of the sensors; and
determining, by the computer system, a final value corresponding to the characteristic based at least in part on a weighted average of the plurality of values.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
determining, by the computer system based on the data received from one or more of the plurality of the sensors, a movement path within the automated-checkout store associated with the person.

### Claim 7 (depends on 1)

The method of claim 1, wherein the determining an interaction between the person and the product item comprises:
determining, based on the data received from one or more of the plurality of the sensors, that the person is located in proximity to the product item in a period of time;
detecting one or more movements of a hand of the person with respect to the product item; and
determining the interaction based on the one or more detected movements.

### Claim 8

A system for tracking a product item in an automated-checkout store comprising a computer system and a plurality of sensors including at least one or more cameras, one or more weight sensors, one or more pressure sensors, and one or more capacitive sensors, the computer system comprising a processor and a non-transitory computer-readable storage medium storing instructions executable by the processor to cause the system to perform operations comprising:
receiving, by the computer system from the plurality of the sensors, data collected by the plurality of the sensors;
identifying, by the computer system, an account associated with a person who enters the store;
receiving, by the computer system, data on movement of the product item collected by the plurality of the sensors;
determining, by the computer system, an interaction between the person and the product item based on the data collected by the plurality of the sensors;
extracting, by the computer system, from the data received from the one or more plurality of the sensors, a plurality of features associated with the product item, wherein the features comprise a conductivity, a color, a position, and a surface force image, the conductivity is used to determine a type of material that makes up at least a part of the product, and the surface force image is used to determine a footprint of the product that includes a shape of a bottom surface of the product;
determining, by the computer system, using a machine-learning model, an identity of the product item based on the extracted features including the conductivity, the color, the position, and the surface force image, wherein the machine-learning model is trained based on training data collected from the plurality of the sensors;
storing, by the computer system, information associating the product item with the person; and
deducting, by the computer system, a payment amount from the identified account, wherein the payment amount is based on a price of the product item associated with the person.

### Claim 9 (depends on 8)

The system of claim 8, wherein one or more of the sensors are affixed to one or more structures in the automated-checkout store, the one or more structures comprising any of:
a ceiling;
a floor;
a shelf;
a rack; and
a refrigerator.

### Claim 10 (depends on 8)

The system of claim 8, wherein the plurality of sensors further comprise one or more of:
a force sensor;
a vibration sensor;
a proximity sensor; and
a resistance-based film sensor.

### Claim 11 (depends on 8)

The system of claim 8, wherein the operations further comprise:
determining, by the computer system based on the data received from a first sensor of the plurality of the sensors, a first identity for the product item associated with a first confidence score;
determining, by the computer system based on the data received from a second sensor of the plurality of the sensors, a second identity for the product item associated with a second confidence score; and
selecting one of the first identity and the second identity as the identity of the product item based on a comparison of the first confidence score and the second confidence score.

### Claim 12 (depends on 8)

The system of claim 8, wherein the operations further comprise:
obtaining, by the computer system based on the data received from the plurality of the sensors, a plurality of values associated with a characteristic of the product item, wherein each of the values is determined based on the data received from one of the plurality of the sensors; and
determining, by the computer system, a final value corresponding to the characteristic based at least in part on a weighted average of the plurality of values.

### Claim 13 (depends on 8)

The system of claim 8, wherein the operations further comprise:
determining, by the computer system based on the data received from one or more of the plurality of the sensors, a movement path within the automated-checkout store associated with the person.

### Claim 14 (depends on 8)

The system of claim 8, wherein the determining an interaction between the person and the product item comprises:
determining, based on the data received from one or more of the plurality of the sensors, that the person is located in proximity to the product item in a period of time;
detecting one or more movements of a hand of the person with respect to the product item; and
determining the interaction based on the one or more detected movements.

### Claim 15

A non-transitory computer-readable storage medium for tracking a product item in an automated-checkout store, configured with instructions executable by one or more processors to cause the one or more processors to perform operations comprising:
receiving, from a plurality of sensors including at least one or more cameras, one or more weight sensors, one or more pressure sensors, and one or more capacitive sensors, data collected by the sensors;
identifying an account associated with a person who enters the store;
receiving, by the computer system, data on movement of the product item collected by the plurality of the sensors;
determining an interaction between the person and the product item based on the data collected by the plurality of the sensors;
extracting, by the computer system, from the data received from the plurality of the sensors, a plurality of features associated with the product item, wherein the features comprise a conductivity, a color, a position, and a surface force image, the conductivity is used to determine a type of material that makes up at least a part of the product, and the surface force image is used to determine a footprint of the product that includes a shape of a bottom surface of the product;
determining, by the computer system, using a machine-learning model, an identity of the product item based on the extracted features including the conductivity, the color, the position, and the surface force image, wherein the machine-learning model is trained based on training data collected from the plurality of the sensors;
storing information associating the product item with the person; and
deducting a payment amount from the identified account, wherein the payment amount is based on a price of the product item associated with the person.

### Claim 16 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the operations further comprise:
determining, by the computer system based on the data received from one or more of the plurality of the sensors, a movement path within the automated-checkout store associated with the person.

### Claim 17 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the determining an interaction between the person and the product item comprises:
determining, based on the data received from the plurality of the sensors, that the person is located in proximity to the product item in a period of time;
detecting one or more movements of a hand of the person with respect to the product item; and
determining the interaction based on the one or more detected movements.

### Claim 18 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the plurality of sensors further comprise any of:
a force sensor;
a vibration sensor;
a proximity sensor; and
a resistance-based film sensor.

### Claim 19 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the operations further comprise:
determining, by the computer system based on the data received from a first sensor of the plurality of the sensors, a first identity for the product item associated with a first confidence score;
determining, by the computer system based on the data received from a second sensor of the plurality of the sensors, a second identity for the product item associated with a second confidence score; and
selecting one of the first identity and the second identity as the identity of the product item based on a comparison of the first confidence score and the second confidence score.

### Claim 20 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the operations further comprise:
obtaining, by the computer system based on the data received from the plurality of the sensors, a plurality of values associated with a characteristic of the product item, wherein each of the values is determined based on the data received from one of the plurality of the sensors; and
determining, by the computer system, a final value corresponding to the characteristic based at least in part on a weighted average of the plurality of values.