# US 2024/0202694 A1 · Subregion transformation for label decoding by an automated checkout system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202694 A1
- Document ID: US20240202694A1
- Status: Published application
- Kind: A1
- Application number: 18/169,012
- Filed: 2023-02-14
- Published: 2024-06-20
- Earliest claimed: 2022-12-19
- Assignees: Maplebear, Inc.(dba Instacart)
- Applicants: Maplebear Inc. (dba Instacart)
- Inventors: Ganglu Wu; Shiyuan Yang; Xiao Zhou; Qi Wang; Qunwei Liu; Youming Luo
- Web page: https://patentgrep.com/patent/US20240202694A1

## CPC

- G06K 7/1413: {1D bar codes}
- G06K 7/1443: {locating of the code in an image}
- G06Q 20/208: {Input by product or record sensing, e.g. weighing or scanner processing}
- G06Q 30/0633: {Managing shopping lists, e.g. compiling or processing purchase lists (shipping orders G06Q10/083; order filling G06Q10/087)}
- G06Q 30/0641: {utilising user interfaces specially adapted for shopping}
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00)
- G06V 10/247: {by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids}
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06V 10/32: Normalisation of the pattern dimensions
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 2201/07: Target detection
- G06V 2201/10: Recognition assisted with metadata
- G06K: GRAPHICAL DATA READING (image or video recognition or understanding G06V); PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An automated checkout system modifies received images of machine-readable labels to improve the performance of a label detection model that the system uses to decode item identifiers encoded in the machine-readable labels. For example, the automated checkout system may transform subregions of an image of a machine-readable label to adjust for distortions in the image's depiction of the machine-readable label. Similarly, the automated checkout system may identify readable regions within received images of machine-readable labels and apply a label detection model to those readable regions. By modifying received images of machine-readable labels, these techniques improve on existing computer-vision technologies by allowing for the effective decoding of machine-readable labels based on real-world images using relatively clean training data.

## Claims

### Claim 1

A method comprising, at a computer system comprising a processor and a computer-readable medium:
receiving an image depicting a machine-readable label;
identifying a readable region of the image by applying a readability detection model to the image, wherein the readability detection model is a model that is trained to identify regions within images depicting machine-readable labels that are likely to be identifiable by a label decoding model, wherein the label decoding model is a machine-learning model that is trained to identify an item identifier encoded in machine-readable labels based on images of the machine-readable labels, wherein training the readability detection model comprises:
receiving a training image of a known machine-readable label, wherein the known machine-readable label is a machine-readable label encoding a known item identifier;
generating a test region on the training image, wherein the test region comprises a portion of the training image;
applying the label decoding model to the test region to determine whether the label detection model correctly identifies the known item identifier; and
responsive to the label detection model correctly identifying the known item identifier, generating a training example for the readability detection model based on the training image and the test region;
resizing the readable region of the image to dimensions for input to the label decoding model; and
identifying an item identifier encoded in the machine-readable label depicted in the received image by applying the label decoding model to the resized readable region of the image.

### Claim 2 (depends on 1)

The method of claim 1, wherein the image is received from a camera attached to a shopping cart, and wherein the method further comprises:
transmitting the item identifier to the shopping cart for display to a user operating the shopping cart via a user interface coupled to the shopping cart.

### Claim 3 (depends on 1)

The method of claim 1, wherein the image is received from a client device operated by a user, and wherein the method further comprises:
transmitting the item identifier to the client device for display to a user operating the client device.

### Claim 4 (depends on 1)

The method of claim 1, wherein applying the label decoding model to the test region comprises:
resizing the portion of the training image corresponding to the test region to dimensions for input to the label decoding model.

### Claim 5 (depends on 1)

The method of claim 1, wherein generating a training example based on the training image and the test region comprises:
generating a training example that comprises the training image of the known machine-readable label and a label comprising the test region.

### Claim 6 (depends on 1)

The method of claim 1, wherein training the readability detection model further comprises:
responsive to the label detection model predicting that the known machine-readable label encodes an item identifier that is not the known item identifier, generating a training example that comprises the training image of the known machine-readable label and a label comprising the test region and an indication that the test region is not readable.

### Claim 7 (depends on 1)

The method of claim 1, wherein training the readability detection model further comprises:
responsive to the label detection model computing a confidence score for an item identifier prediction that does not exceed a threshold value, generating a training example that comprises the training image of the known machine-readable label and a label comprising the test region and an indication that the test region is not readable.

### Claim 8 (depends on 1)

The method of claim 1, wherein resizing the readable region of the image comprises:
upsampling the portion of the received image corresponding to the readable region.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
adding an item corresponding to the item identifier encoded in the machine-readable label to a shopping list for a user.

### Claim 10 (depends on 9)

The method of claim 9, wherein the received image further depicts the machine-readable label affixed to the item corresponding to the item identifier.

### Claim 11

A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to:
receive an image depicting a machine-readable label;
identify a readable region of the image by applying a readability detection model to the image, wherein the readability detection model is a model that is trained to identify regions within images depicting machine-readable labels that are likely to be identifiable by a label decoding model, wherein the label decoding model is a machine-learning model that is trained to identify an item identifier encoded in machine-readable labels based on images of the machine-readable labels, wherein training the readability detection model comprises:
receiving a training image of a known machine-readable label, wherein the known machine-readable label is a machine-readable label encoding a known item identifier;
generating a test region on the training image, wherein the test region comprises a portion of the training image;
applying the label decoding model to the test region to determine whether the label detection model correctly identifies the known item identifier; and
responsive to the label detection model correctly identifying the known item identifier, generating a training example for the readability detection model based on the training image and the test region;
resize the readable region of the image to dimensions for input to the label decoding model; and
identify an item identifier encoded in the machine-readable label depicted in the received image by applying the label decoding model to the resized readable region of the image.

### Claim 12 (depends on 11)

The computer-readable medium of claim 11, wherein the image is received from a camera attached to a shopping cart, and wherein the computer-readable medium further stores instructions that cause a processor to:
transmit the item identifier to the shopping cart for display to a user operating the shopping cart via a user interface coupled to the shopping cart.

### Claim 13 (depends on 11)

The computer-readable medium of claim 11, wherein the image is received from a client device operated by a user, and wherein the computer-readable medium further stores instructions that cause a processor to:
transmit the item identifier to the client device for display to a user operating the client device.

### Claim 14 (depends on 11)

The computer-readable medium of claim 11, wherein the instructions for applying the label decoding model to the test region comprise instructions that cause a processor to:
resize the portion of the training image corresponding to the test region to dimensions for input to the label decoding model.

### Claim 15 (depends on 11)

The computer-readable medium of claim 11, wherein the instructions for generating a training example based on the training image and the test region comprise instructions that cause a processor to:
generate a training example that comprises the training image of the known machine-readable label and a label comprising the test region.

### Claim 16 (depends on 11)

The computer-readable medium of claim 11, wherein the instructions for training the readability detection model further comprise instructions that cause a processor to:
responsive to the label detection model predicting that the known machine-readable label encodes an item identifier that is not the known item identifier, generate a training example that comprises the training image of the known machine-readable label and a label comprising the test region and an indication that the test region is not readable.

### Claim 17 (depends on 11)

The computer-readable medium of claim 11, wherein the instructions for training the readability detection model further comprise instructions that cause a processor to:
responsive to the label detection model computing a confidence score for an item identifier prediction that does not exceed a threshold value, generate a training example that comprises the training image of the known machine-readable label and a label comprising the test region and an indication that the test region is not readable.

### Claim 18 (depends on 11)

The computer-readable medium of claim 11, wherein the instructions for resizing the readable region of the image comprise instructions that cause a processor to:
upsample the portion of the received image corresponding to the readable region.

### Claim 19 (depends on 11)

The computer-readable medium of claim 11, further storing instructions that cause a processor to:
add an item corresponding to the item identifier encoded in the machine-readable label to a shopping list for a user.

### Claim 20

A system comprising:
a processor
non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the processor to:
receive an image depicting a machine-readable label;
identify a readable region of the image by applying a readability detection model to the image, wherein the readability detection model is a model that is trained to identify regions within images depicting machine-readable labels that are likely to be identifiable by a label decoding model, wherein the label decoding model is a machine-learning model that is trained to identify an item identifier encoded in machine-readable labels based on images of the machine-readable labels, wherein training the readability detection model comprises:
receiving a training image of a known machine-readable label, wherein the known machine-readable label is a machine-readable label encoding a known item identifier;
generating a test region on the training image, wherein the test region comprises a portion of the training image;
applying the label decoding model to the test region to determine whether the label detection model correctly identifies the known item identifier; and
responsive to the label detection model correctly identifying the known item identifier, generating a training example for the readability detection model based on the training image and the test region;
resize the readable region of the image to dimensions for input to the label decoding model; and
identify an item identifier encoded in the machine-readable label depicted in the received image by applying the label decoding model to the resized readable region of the image.
