# US 2023/0410185 A1 · Self-checkout system, purchased product management method, and purchased product management program

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410185 A1
- Document ID: US20230410185A1
- Status: Published application
- Kind: A1
- Application number: 18/238,029
- Filed: 2023-08-25
- Published: 2023-12-21
- Earliest claimed: 2020-08-21
- Assignees: NEC Corporation
- Applicants: NEC Corporation
- Inventors: Kyota Higa; Emi Kitagawa
- Web page: https://patentgrep.com/patent/US20230410185A1

## CPC

- G06Q 30/0633: {Managing shopping lists, e.g. compiling or processing purchase lists (shipping orders G06Q10/083; order filling G06Q10/087)}
- G06Q 10/087: Inventory or stock management, e.g. order filling, procurement or balancing against orders
- G06Q 10/08724: {by shelf level inventory management, e.g. planograms}
- G06Q 20/18: involving self-service terminals [SST], vending machines, kiosks or multimedia terminals
- G06Q 20/20: Point-of-sale [POS] network systems
- G06Q 20/201: {Price look-up processing, e.g. updating}
- G06Q 20/203: {Inventory monitoring}
- G06Q 20/208: {Input by product or record sensing, e.g. weighing or scanner processing}
- G06Q 30/06: Buying, selling or leasing transactions
- G06T 7/0008: {checking presence/absence}
- G06T 7/254: involving subtraction of images
- G06T 7/74: {involving reference images or patches}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G07G 1/0036: {Checkout procedures}
- G07G 1/14: Systems including one or more distant stations co-operating with a central processing unit
- G06Q 10/0877: {by inventory control or reporting using inventory tracking or counting}
- G06T 2207/10024: Color image
- G06T 2207/30128: Food products
- G06T 2207/30196: Human being; Person
- G06T 2207/30232: Surveillance
- G06T 2207/30242: Counting objects in image
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G07G: REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS

## Abstract

A change detection means 840 detects an area of change in a product shelf on which a product has been arranged, on the basis of a captured image of the product. A classification means 850 classifies the change in the product shelf in the detected area of change. A shopping list generation means 860 generates a shopping list for a customer on the basis of the classification of the change in the product shelf and of shelving information on the product shelf.

## Claims

### Claim 1

A self-checkout system comprising a hardware processor configured to execute a software code to:
detect an area of change in a product shelf on which a product has been arranged, based on a captured image of the product;
classify the change in the product shelf in the detected area of change; and
generate a shopping list for a customer based on the classification of the change in the product shelf and of shelving information on the product shelf,
wherein the classification of the change in the product shelf is one of a change caused by a product having been taken, a change in a store environment of the product shelf, a change caused by a product having been arranged on the product shelf, and a change caused by a different appearance of the product, and
wherein the change caused by a different appearance of the product, which is the classification of the change in the product shelf, includes a change in appearance caused by a different product having been arranged and a change in appearance caused by a change in product attitude.

### Claim 2 (depends on 1)

The self-checkout system according to claim 1, wherein the hardware processor is configured to execute the software code to register the product on the shopping list in a case where the classification of the change in the product shelf is a change caused by a product having been taken.

### Claim 3 (depends on 1)

The self-checkout system according to claim 1, wherein the hardware processor is configured to execute the software code to delete the product from the shopping list in a case where the classification of the change in the product shelf is a change caused by a product having been arranged on the product shelf.

### Claim 4 (depends on 1)

The self-checkout system according to claim 1, wherein the hardware processor is configured to execute the software code to:
generate relationship information indicating a relationship between an area of change and a person by associating the area of change with the person, based on the imaging time and person information acquired from the captured image and based on the position information and the imaging time of the area of change acquired from the captured image; and
generate the shopping list corresponding to the person based on the classification of the change in the product shelf in the area of change and of the shelving information on the product shelf.

### Claim 5 (depends on 1)

The self-checkout system according to claim 1, wherein the hardware processor is configured to execute the software code to:
generate relationship information indicating a relationship between an area of change and a person by associating the area of change with the person, based on the information acquisition time and position information of the person detected by a flow line detection unit for detecting a flow line of a person in a store and based on the position information and imaging time of the area of change acquired from the captured image; and
generate a shopping list corresponding to the person based on the classification of the change in the product shelf in the area of change and of the shelving information on the product shelf.

### Claim 6 (depends on 1)

The self-checkout system according to claim 1, wherein the hardware processor is configured to execute the software code to classify the change in the product shelf from the image of the area of change by using a shelf change model that is a model of a change in the product shelf depending on before and after a time change.

### Claim 7 (depends on 1)

The self-checkout system according to claim 1, wherein the hardware processor is configured to execute the software code to detect the area of change in the product shelf on which the product has been arranged on based on an RGB image and a range image obtained by imaging the imaging range of the RGB image as the captured images of the product.

### Claim 8

A purchased product management method comprising:
detecting an area of change in a product shelf on which a product has been arranged, based on a captured image of the product;
classifying the change in the product shelf in the detected area of change; and
generating a shopping list for a customer based on the classification of the change in the product shelf and of shelving information on the product shelf,
wherein the classification of the change in the product shelf is one of a change caused by a product having been taken, a change in a store environment of the product shelf, a change caused by a product having been arranged on the product shelf, and a change caused by a different appearance of the product, and
wherein the change caused by a different appearance of the product, which is the classification of the change in the product shelf, includes a change in appearance caused by a different product having been arranged and a change in appearance caused by a change in product attitude.

### Claim 9

A non-transitory computer readable information recording medium storing a purchased product management program, when executed by a processor, that performs a method for:
detecting an area of change in a product shelf on which a product has been arranged, based on a captured image of the product;
classifying the change in the product shelf in the detected area of change; and
generating a shopping list for a customer based on the classification of the change in the product shelf and of shelving information on the product shelf,
wherein the classification of the change in the product shelf is one of a change caused by a product having been taken, a change in a store environment of the product shelf, a change caused by a product having been arranged on the product shelf, and a change caused by a different appearance of the product, and
wherein the change caused by a different appearance of the product, which is the classification of the change in the product shelf, includes a change in appearance caused by a different product having been arranged and a change in appearance caused by a change in product attitude.
