# US 2023/0267292 A1 · Subregion transformation for label decoding by an automated checkout system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0267292 A1
- Document ID: US20230267292A1
- Status: Published application
- Kind: A1
- Application number: 18/169,010
- Filed: 2023-02-14
- Published: 2023-08-24
- Earliest claimed: 2022-12-19
- Assignees: Maplebear, Inc.(dba Instacart)
- Applicants: Maplebear Inc. (dba Instacart)
- Inventors: Ganglu Wu; Shiyuan Yang; Xiao Zhou; Qi Wang; Qunwei Liu; Youming Luo
- Web page: https://patentgrep.com/patent/US20230267292A1

## CPC

- G06K 7/1413: {1D bar codes}
- G06K 7/1443: {locating of the code in an image}
- G06Q 20/208: {Input by product or record sensing, e.g. weighing or scanner processing}
- G06Q 30/0633: {Managing shopping lists, e.g. compiling or processing purchase lists (shipping orders G06Q10/083; order filling G06Q10/087)}
- G06Q 30/0641: {utilising user interfaces specially adapted for shopping}
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00)
- G06V 10/247: {by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids}
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06V 10/32: Normalisation of the pattern dimensions
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 2201/07: Target detection
- G06V 2201/10: Recognition assisted with metadata
- G06K: GRAPHICAL DATA READING (image or video recognition or understanding G06V); PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An automated checkout system modifies received images of machine-readable labels to improve the performance of a label detection model that the system uses to decode item identifiers encoded in the machine-readable labels. For example, the automated checkout system may transform subregions of an image of a machine-readable label to adjust for distortions in the image's depiction of the machine-readable label. Similarly, the automated checkout system may identify readable regions within received images of machine-readable labels and apply a label detection model to those readable regions. By modifying received images of machine-readable labels, these techniques improve on existing computer-vision technologies by allowing for the effective decoding of machine-readable labels based on real-world images using relatively clean training data.

## Claims

### Claim 1

A method comprising, at a computer system comprising a processor and a computer-readable medium:
receiving an image depicting a machine-readable label;
generating a set of boundary points on the image by applying a first machine-learning model to the image, wherein the first machine-learning model is trained to identify boundary points on an image;
generating a set of initial subregions of the image based on the boundary points, wherein each initial subregion of the set of initial subregions comprises a plurality of corners at a subset of the set of boundary points;
generating a transformation matrix for each initial subregion of the set of initial subregions, wherein each transformation matrix comprises a set of values for transforming a corresponding initial subregion into a standardized subregion;
generating a set of standardized subregions of the image by applying each transformation matrix to a corresponding initial subregion of the set of initial subregions;
concatenating the set of standardized subregions into a modified image; and
identifying an item identifier encoded in the machine-readable label by applying a label decoding model to the modified image, wherein the label decoding model is trained to predict item identifiers based on images of machine-readable labels.

### Claim 2 (depends on 1)

The method of claim 1, wherein the image is received from a camera attached to a shopping cart, and wherein the method further comprises:
transmitting the item identifier to the shopping cart for display to a user operating the shopping cart via a user interface coupled to the shopping cart.

### Claim 3 (depends on 1)

The method of claim 1, wherein the image is received from a client device operated by a user, and wherein the method further comprises:
transmitting the item identifier to the client device for display to a user operating the client device.

### Claim 4 (depends on 1)

The method of claim 1, wherein generating a set of boundary points on the image comprises:
identifying a plurality of points in the image depicting plurality of corners of the machine-readable label; and
generating a plurality of boundary points based on the plurality of points depicting the plurality of corners.

### Claim 5 (depends on 4)

The method of claim 4, wherein generating a set of boundary points on the image comprises:
identifying a point on the image between a pair of the plurality of points depicting the plurality of corners; and
generating a boundary point based on the identified point.

### Claim 6 (depends on 1)

The method of claim 1, wherein each of the set of initial subregions of the image is a quadrilateral.

### Claim 7 (depends on 6)

The method of claim 6, wherein each of the set of standardized subregions is a rectangle.

### Claim 8 (depends on 1)

The method of claim 1, wherein each of the set of initial subregions of the image is a triangle.

### Claim 9 (depends on 8)

The method of claim 8, wherein each of the set of standardized subregions is one of an equilateral triangle, an isosceles triangle, or a right triangle.

### Claim 10 (depends on 1)

The method of claim 1, wherein each of the initial subregions of the image is an irregular shape.

### Claim 11

A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to:
receive an image depicting a machine-readable label;
generate a set of boundary points on the image by applying a first machine-learning model to the image, wherein the first machine-learning model is trained to identify boundary points on an image;
generate a set of initial subregions of the image based on the boundary points, wherein each initial subregion of the set of initial subregions comprises a plurality of corners at a subset of the set of boundary points;
generate a transformation matrix for each initial subregion of the set of initial subregions, wherein each transformation matrix comprises a set of values for transforming a corresponding initial subregion into a standardized subregion;
generate a set of standardized subregions of the image by applying each transformation matrix to a corresponding initial subregion of the set of initial subregions;
concatenate the set of standardized subregions into a modified image; and
identify an item identifier encoded in the machine-readable label by applying a label decoding model to the modified image, wherein the label decoding model is trained to predict item identifiers based on images of machine-readable labels.

### Claim 12 (depends on 11)

The computer-readable medium of claim 11, wherein the image is received from a camera attached to a shopping cart, and wherein the computer-readable medium further stores instructions that cause a processor to:
transmit the item identifier to the shopping cart for display to a user operating the shopping cart via a user interface coupled to the shopping cart.

### Claim 13 (depends on 11)

The computer-readable medium of claim 11, wherein the image is received from a client device operated by a user, and wherein the computer-readable medium further stores instructions that cause a processor to:
transmit the item identifier to the client device for display to a user operating the client device.

### Claim 14 (depends on 11)

The computer-readable medium of claim 11, wherein the instructions for generating a set of boundary points on the image comprise instructions that, when executed, cause a processor to:
identify a plurality of points in the image depicting plurality of corners of the machine-readable label; and
generate a plurality of boundary points based on the plurality of points depicting the plurality of corners.

### Claim 15 (depends on 14)

The computer-readable medium of claim 14, wherein the instructions for generating a set of boundary points on the image comprise instructions that, when executed, cause a processor to:
identify a point on the image between a pair of the plurality of points depicting the plurality of corners; and
generate a boundary point based on the identified point.

### Claim 16 (depends on 11)

The computer-readable medium of claim 11, wherein each of the set of initial subregions of the image is a quadrilateral.

### Claim 17 (depends on 16)

The computer-readable medium of claim 16, wherein each of the set of standardized subregions is a rectangle.

### Claim 18 (depends on 11)

The computer-readable medium of claim 11, wherein each of the set of initial subregions of the image is a triangle.

### Claim 19 (depends on 18)

The computer-readable medium of claim 18, wherein each of the set of standardized subregions is one of an equilateral triangle, an isosceles triangle, or a right triangle.

### Claim 20

A system comprising:
a processor; and
a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the processor to:
receive an image depicting a machine-readable label;
generate a set of boundary points on the image by applying a first machine-learning model to the image, wherein the first machine-learning model is trained to identify boundary points on an image;
generate a set of initial subregions of the image based on the boundary points, wherein each initial subregion of the set of initial subregions comprises a plurality of corners at a subset of the set of boundary points;
generate a transformation matrix for each initial subregion of the set of initial subregions, wherein each transformation matrix comprises a set of values for transforming a corresponding initial subregion into a standardized subregion;
generate a set of standardized subregions of the image by applying each transformation matrix to a corresponding initial subregion of the set of initial subregions;
concatenate the set of standardized subregions into a modified image; and
identify an item identifier encoded in the machine-readable label by applying a label decoding model to the modified image, wherein the label decoding model is trained to predict item identifiers based on images of machine-readable labels.
