# US 2021/0279784 A1 · Analysis method and system for the item on the supermarket shelf

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279784 A1
- Document ID: US20210279784A1
- Status: Published application
- Kind: A1
- Application number: 16/764,059
- Filed: 2020-03-09
- Published: 2021-09-09
- Earliest claimed: 2020-03-09
- Assignees: Yi Tunnel (Beijing) Technology Co., Ltd.
- Applicants: Yi Tunnel (Beijing) Technology Co., Ltd.
- Inventors: Yili WU
- Web page: https://patentgrep.com/patent/US20210279784A1

## CPC

- G06Q 30/0283: Price estimation or determination
- G06Q 30/0623: {by investigating goods or services}
- G06T 7/0008: {checking presence/absence}
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/82: using neural networks
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 30/18057: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 30/19173: {Classification techniques}
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30128: Food products
- G06T 2207/30232: Surveillance
- G06V 20/68: Food, e.g. fruit or vegetables
- G06V 30/10: Character recognition
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The present invention belongs to the technical field of visual identification, and discloses an analysis method and system for items on the supermarket shelf. The method comprises acquiring one shelf image, which contains the items on the supermarket shelf and price tags corresponding to the items, and one shelf image corresponds to one shooting angle; acquiring a primary item classification result corresponding to the items on the supermarket shelf and a primary price identification result corresponding to the price tag according to the shelf image, a pre-trained primary item classification model and price tag text identification model; displaying the primary item classification result and the primary price identification result on the shelf image. The system comprises an image acquisition device, a primary classification device and a display device. According to the above technical solutions, the present invention prevents the managers of the shopping place from consulting the item packaging one by one in front of the shelf when managing the item information on the shelf, and then recording the item information, thus improving the management efficiency and facilitating the managers to know all the item information on the supermarket shelf at a glance.

## Claims

### Claim 1

An analysis method for items on a supermarket shelf comprising:
a step of image acquisition acquiring one shelf image, wherein the shelf image contains items on the supermarket shelf and price tags corresponding to the items, and one shelf image corresponds to one shooting angle;
a step of primary classification acquiring a primary item classification result corresponding to the items on the supermarket shelf and a primary price identification result corresponding to the price tags according to the shelf image, a pre-trained primary item classification model and a pre-trained price tag text identification model, the primary item classification model is a model that is constructed by an image identification technique of a convolutional neural network and trained by all the items on the supermarket shelf, and the price tag text identification model is a model that is constructed by the image identification technique of the convolutional neural network and trained by the price tags corresponding to all the items on the supermarket shelf;
a step of display displaying the primary item classification result and the primary price identification result on the shelf image.

### Claim 2 (depends on 1)

The analysis method according to claim 1, wherein after the step of acquiring the primary item classification result corresponding to the items on the supermarket shelf, the analysis method further comprises:
acquiring an item price corresponding to the primary item classification result;
judging whether the item price is consistent with the primary price identification result or not, and if not, sending a first reminder message.

### Claim 3 (depends on 1)

The analysis method according to claim 1, when the items on the supermarket shelf are fruit and fresh-items, after the step of primary classification, the analysis method further comprises:
judging whether the primary item classification result is in a decayed state or not according to the shelf image and a pre-trained fruit and fresh-item decay model corresponding to the primary item classification result;
if the primary item classification result is in the decayed state, sending a second reminder message;
wherein the fruit and fresh-item decay model is a model that is constructed by the image identification technique of the convolutional neural network and trained by all the items in the decayed state on the supermarket shelf.

### Claim 4 (depends on 1)

The analysis method according to claim 1, wherein after the step of primary classification, the analysis method further comprises:
judging whether the items on the supermarket shelf are in a sold-out state or not according to the shelf image and a pre-trained empty shelf model;
if the items on the supermarket shelf are in a sold-out state, sending a third reminder message.

### Claim 5 (depends on 1)

The analysis method according to claim 1, wherein before the step of image acquisition, the analysis method further comprises:
judging whether the number of the shooting angles is a plurality, and the plurality of shooting angles correspond to a plurality of the shelf images one by one;
if so, sequentially executing the step of primary classification to each shelf image of the acquired plurality of shelf images to obtain the primary item classification result corresponding to the items on the supermarket shelf and the primary price identification result corresponding to the price tags in each shelf image, and acquiring a first-level item classification result and a first-level price identification result according to a plurality of primary item classification results, a plurality of primary price identification results, a pre-trained first-level item linear regression model and a pre-trained first-level price tag linear regression model, and displaying the first-level item classification result and the first-level price identification result on the shelf image;
if not, skipping to the step of image acquisition.

### Claim 6 (depends on 5)

The analysis method according to claim 5, wherein after acquiring the first-level item classification result and the first-level price identification result according to the plurality of primary item classification results, the plurality of primary price identification results and the pre-trained first-level linear regression model, the analysis method further comprises:
judging whether the primary item classification result is a similar item or not;
if the primary item classification result is a similar item, acquiring a secondary item classification result corresponding to the similar item according to the plurality of shelf images and a pre-trained second-level item classification model, and then acquiring a second-level item classification result corresponding to the similar item according to the plurality of secondary item classification results and a pre-trained second-level item linear regression model, the second-level classification model is a model that is constructed by the image identification technique of the convolutional neural network in advance and trained by all the similar items on the supermarket shelf;
correspondingly, displaying the second-level item classification result and the first-level price identification result on the shelf image;
if not, skipping to the step of display.

### Claim 7

An analysis system for items on a supermarket shelf comprising:
an image acquisition device used to acquire one shelf image, and the shelf image contains items on the supermarket shelf and price tags corresponding to the items, and one shelf image corresponds to one shooting angle;
a primary classification device used to acquire a primary item classification result corresponding to the items on the supermarket shelf and a primary price identification result corresponding to the price tag according to the shelf image, a pre-trained primary item classification model and a pre-trained price tag text identification model, the primary item classification model is a model that is constructed by an image identification technique of a convolutional neural network and trained by all the items on the supermarket shelf, and the price tag text identification model is a model that is constructed by the image identification technique of the convolutional neural network and trained by the price tags corresponding to all the items on the supermarket shelf;
a display device used to display the primary item classification result and the primary price identification result on the shelf image.

### Claim 8 (depends on 7)

The analysis system according to claim 7, further comprising:
a first reminder module used to acquire an item price corresponding to the item classification result, and judge whether the item price is consistent with the primary price identification result or not, and if not, send a first reminder message.

### Claim 9 (depends on 7)

The analysis system according to claim 7, when the items on the supermarket shelf are fruit and fresh-items, the analysis system further comprising:
a second reminder module used to judge whether the primary item classification result is in a decayed state or not according to the shelf image and a pre-trained fruit and fresh-item decay model corresponding to the primary item classification result, and if the primary item classification result is in a decayed state, send a second reminder message, wherein the fruit and fresh-item decay model is a model that is constructed by the image identification technique of the convolutional neural network and trained by all the items in the decayed state on the supermarket shelf.

### Claim 10 (depends on 7)

The analysis system according to claim 7, further comprising:
a third reminder module used to judge whether the items on the supermarket shelf are in a sold-out state or not according to the shelf image and a pre-trained empty shelf model, and if he items on the supermarket shelf are in a sold-out state, send a third reminder message.
