# US 2021/0174299 A1 · Method, system and device for collecting sales information of commodities in cabinet, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174299 A1
- Document ID: US20210174299A1
- Status: Published application
- Kind: A1
- Application number: 17/116,361
- Filed: 2020-12-09
- Published: 2021-06-10
- Earliest claimed: 2019-12-09
- Assignees: Shanghai Clobotics Technology Co., Ltd.
- Applicants: Shanghai Clobotics Technology Co., Ltd.
- Inventors: Xing Tong; Liang Sang; Yan Ke; Peizhao Li
- Web page: https://patentgrep.com/patent/US20210174299A1

## CPC

- G06Q 10/0875: Itemisation or classification of parts, supplies or services, e.g. bill of materials
- G06N 20/00: Machine learning
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

The present disclosure relates to the field of image processing technology, in particular to a method, system and device for collecting sales information of commodities in a cabinet, and a storage medium. Steps of the method include: successively acquiring pictures of the cabinet with the commodities placed therein in chronological order; acquiring commodity information on the former picture of every two adjacent cabinet pictures; successively adjusting every two adjacent cabinet pictures; comparing the corresponding commodity areas in every two adjacent cabinet pictures after the adjustment; and in the presence of different image contents, acquiring the number of commodity areas with different image contents and corresponding commodity information on the former picture. In the present disclosure, the traditional method of manually collecting commodity sales information is changed to an automatic collection method, thereby greatly improving the collection efficiency of commodity sales information, and reducing manual labor intensity, and also dynamic changes of commodity sales can be known in real time.

## Claims

### Claim 1

A method for collecting sales information of commodities in a cabinet, comprising the following steps:
successively acquiring pictures of the cabinet with the commodities placed therein in chronological order;
acquiring commodity information on the former picture of every two adjacent cabinet pictures;
successively adjusting every two adjacent cabinet pictures, so that photographing angles and positions of commodities in corresponding commodity areas of the two adjacent cabinet pictures are consistent;
comparing the corresponding commodity areas in every two adjacent cabinet pictures after the adjustment to determine whether different image contents are present in the corresponding commodity areas; and
in the presence of different image contents, acquiring the number of commodity areas with different image contents and corresponding commodity information on the former picture.

### Claim 2 (depends on 1)

The method for collecting sales information of commodities in a cabinet according to claim 1, wherein the step of successively adjusting every two adjacent cabinet pictures, so that photographing angles and positions of commodities in corresponding commodity areas of the two pictures are consistent comprises:
acquiring feature points on different commodities in the two adjacent cabinet pictures;
pairing the feature points in the two adjacent cabinet pictures to calculate a homography matrix; and
in the order of photographing time, performing a perspective transformation on the latter picture of the two pictures according to the homography matrix, to obtain two pictures in which the photographing angles and the positions of the commodities in the corresponding commodity areas are consistent.

### Claim 3 (depends on 1)

The method for collecting sales information of commodities in a cabinet according to claim 1, wherein the step of successively adjusting every two adjacent cabinet pictures of a number of pictures, so that photographing angles and positions of commodities in corresponding commodity areas of the two pictures are consistent comprises:
in the order of photographing time, detecting commodities in the former picture of the two adjacent cabinet pictures to provide a first marker box for the commodities in the former picture;
using a template matching method, marking the latter picture with a second marker box corresponding to the first marker box;
using center points of the two corresponding marker boxes as corresponding feature points;
calculating a homography matrix by using the corresponding feature points; and
performing a perspective transformation on the latter picture of the two adjacent pictures by using the homography matrix to obtain two pictures in which the photographing angles and the positions of the commodities in the corresponding commodity areas are consistent.

### Claim 4 (depends on 1)

The method for collecting sales information of commodities in a cabinet according to claim 1, wherein the step of comparing the corresponding commodity areas in every two adjacent cabinet pictures after the adjustment to determine whether different image contents are present at positions in the corresponding commodity areas comprises:
performing channel dimension stacking on every two adjacent cabinet pictures after the adjustment to obtain an image matrix in which the corresponding commodity areas coincide; and
inputting the image matrix into a recognition model capable of recognizing image contents of the corresponding commodity areas in the image matrix, to detect commodity areas with different contents.

### Claim 5 (depends on 4)

The method for collecting sales information of commodities in a cabinet according to claim 4, wherein a method for training the recognition model comprises the following steps:
successively acquiring, in chronological order, a plurality of simulated pictures that simulate changes in the sales of the commodities in the cabinet;
successively adjusting every two adjacent pictures of the plurality of simulated pictures, so that photographing angles and positions of commodities in corresponding commodity areas of the two pictures are consistent;
performing marking in the former picture arranged in chronological order of every two adjacent simulated pictures after the adjustment to mark an area where a sold commodity is simulated;
performing channel dimension stacking on every two adjacent simulated pictures after the marking to obtain a simulated image matrix in which the corresponding commodity areas coincide; and
training a model by using the simulated image matrix to obtain a recognition model capable of recognizing image contents of the corresponding commodity areas.

### Claim 6 (depends on 1)

The method for collecting sales information of commodities in a cabinet according to claim 1, wherein the step of acquiring commodity information on the former picture of every two adjacent cabinet pictures comprises:
detecting all commodities on the former picture of every two adjacent cabinet pictures;
performing cutout processing on an area occupied by each commodity on the former picture, so that the area occupied by each commodity forms an independent cutout image; and
recognizing the cutout image to obtain commodity information in each cutout image in one-to-one correspondence with each commodity on the former picture.

### Claim 7 (depends on 1)

The method for collecting sales information of commodities in a cabinet according to claim 1, wherein the method further comprises:
inputting the former picture of every two adjacent cabinet pictures into a pre-trained repeated area detection model to detect a repeated area in the former picture; and
deducting commodity information and quantity that are double-counted in the repeated area to obtain the number of commodity areas with different image contents and corresponding commodity information after deduplication.

### Claim 8 (depends on 1)

A system for collecting sales information of commodities in a cabinet, for implementing the steps of the method for collecting sales information of commodities in a cabinet according to claim 1, wherein the system comprises:
a data acquisition module for successively acquiring pictures of the cabinet with the commodities placed therein in chronological order; and
acquiring commodity information on the former picture of every two adjacent cabinet pictures;
a data processing module for successively adjusting every two adjacent cabinet pictures, so that photographing angles and positions of commodities in corresponding commodity areas of the two adjacent cabinet pictures are consistent; and
an image recognition module for comparing the corresponding commodity areas in every two adjacent cabinet pictures after the adjustment to determine whether different image contents are present in the corresponding commodity areas; and
in the presence of different image contents, outputting the number of commodity areas with different image contents and corresponding commodity information on the former picture.

### Claim 9 (depends on 1)

A device for collecting sales information of commodities in a cabinet, comprising:
a memory for storing a computer program; and
a processor for implementing the steps of the method for collecting sales information of commodities in a cabinet according to claim 1 when executing the computer program.

### Claim 10 (depends on 1)

A computer readable storage medium, wherein the computer readable storage medium stores a computer program which, when executed by a processor, implements the steps of the method for collecting sales information of commodities in a cabinet according to claim 1.
