# US 2023/0267685 A1 · 3D Product Reconstruction from Multiple Images Collected at Checkout Lanes

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0267685 A1
- Document ID: US20230267685A1
- Status: Published application
- Kind: A1
- Application number: 17/677,686
- Filed: 2022-02-22
- Published: 2023-08-24
- Earliest claimed: 2022-02-22
- Assignees: Zebra Technologies Corporation
- Applicants: Zebra Technologies Corporation
- Inventors: Alessandro Bay; Andrea Mirabile
- Web page: https://patentgrep.com/patent/US20230267685A1

## CPC

- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06K 7/1413: {1D bar codes}
- G06Q 20/208: {Input by product or record sensing, e.g. weighing or scanner processing}
- G06Q 30/0276: {Advertisement creation}
- G06Q 30/0282: Rating or review of business operators or products
- G06Q 30/0643: {graphically representing goods, e.g. 3D product representation}
- G06T 7/194: involving foreground-background segmentation
- G06T 7/337: {involving reference images or patches}
- G06T 7/40: Analysis of texture (depth or shape recovery from texture G06T7/529)
- G06T 7/50: Depth or shape recovery
- G06T 7/55: from multiple images
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/74: {involving reference images or patches}
- G06T 15/04: Texture mapping
- G06V 10/54: relating to texture
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 20/647: {by matching two-dimensional images to three-dimensional objects}
- G07G 1/0063: {with means for detecting the geometric dimensions of the article of which the code is read, such as its size or height, for the verification of the registration}
- G06T 2200/08: involving all processing steps from image acquisition to 3D model generation
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/30244: Camera pose
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06K: GRAPHICAL DATA READING (image or video recognition or understanding G06V); PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G07G: REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS

## Abstract

Techniques for three-dimensional product reconstruction using multiple images collected at checkout lanes are disclosed herein. An example method includes capturing, by a barcode reader associated with a point of sale (POS) workstation, first image data associated with each of a plurality of products passing through a product scanning region of the POS workstation; analyzing barcode data from the first image data captured by the barcode reader to identify each product of the plurality of products passing through the product scanning region of the POS workstation; capturing, by one or more color cameras associated with the POS workstation, second image data associated with the identified product, of the plurality of products passing through the product scanning region of the POS workstation; and generating, by a processor, a textured three-dimensional mesh reconstruction for the identified product based on the second image data associated with the identified product.

## Claims

### Claim 1

A method for three-dimensional product reconstruction using multiple images collected at POS workstations, comprising:
capturing, by a barcode reader associated with a point of sale (POS) workstation, first image data associated with a plurality of products passing through a product scanning region of the POS workstation;
analyzing barcode data from the first image data captured by the barcode reader to identify at least some of the plurality of products passing through the product scanning region of the POS workstation;
responsive to identifying a product, from the plurality of products, resulting in an identified product, capturing, by one or more color cameras associated with the POS workstation, second image data associated with the identified product; and
generating, by a processor, a textured three-dimensional mesh reconstruction for the identified product based on the second image data associated with the identified product.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating the textured three-dimensional mesh reconstruction associated with the identified product based on the second image data associated with the identified product includes:
for each image of the second image data associated with the identified product:
segmenting, by the processor, the identified product from the background, or any hands in the image;
analyzing, by the processor, the segmented image of the identified product to determine a depth and a texture associated with each pixel of the identified product in the image; and
generating, by the processor, a three-dimensional point cloud associated with the segmented image of the identified product based on the depth and the texture associated with each pixel of the identified product; and
stitching, by the processor, the three-dimensional point clouds associated with each image of the second image data associated with the identified product together to generate a textured three-dimensional mesh reconstruction associated with the identified product.

### Claim 3 (depends on 2)

The method of claim 2, wherein stitching, by the processor, the three-dimensional point cloud associated with each image of the second image data associated with the identified product together to generate a textured three-dimensional mesh reconstruction associated with the product includes:
for each three-dimensional point cloud associated with the identified product:
identifying, by the processor, a location of a barcode or other product label in the three-dimensional point cloud; and
determining, by the processor, an orientation of the product in the three-dimensional point cloud based on the identified location of the barcode or other product label in the three-dimensional point cloud; and
stitching, by the processor, the three-dimensional point clouds associated with each image of the second image data associated with the identified product together to generate the textured three-dimensional mesh reconstruction associated with the product, based on the determined orientation of the product in each three-dimensional point cloud.

### Claim 4 (depends on 1)

The method of claim 1, wherein generating the textured three-dimensional mesh reconstruction for the identified product based on the second image data associated with the identified product includes:
determining, by the processor, an indication of a shape associated with the identified product;
generating, by the processor, a three-dimensional mesh associated with the identified product based on the indication of the shape associated with the identified product; and
stitching, by the processor, the second image data associated with the identified product to the three-dimensional mesh associated with the identified product in order to generate a textured three-dimensional mesh associated with the identified product.

### Claim 5 (depends on 4)

The method of claim 4, wherein determining the indication of the shape associated with the identified product includes:
accessing, by the processor, a database storing an indication of a product shape associated with the identified product.

### Claim 6 (depends on 4)

The method of claim 4, wherein determining the indication of the shape associated with the product shown in the image includes:
for at least one image of the second image data associated with the identified product:
segmenting, by the processor, the identified product from the background, or any hands in the image;
analyzing, by the processor, the segmented image of the identified product using a neural network to classify the shape associated with the identified product as one of a plurality of possible product shapes.

### Claim 7 (depends on 4)

The method of claim 4, wherein stitching the second image data associated with the identified product to the three-dimensional mesh associated with the identified product in order to generate the textured three-dimensional mesh associated with the identified product includes:
for each image of the second image data associated with the identified product:
segmenting, by the processor, the identified product from the background, or any hands in the image;
identifying, by the processor, a location of a barcode or other product label in the image; and
determining, by the processor, an orientation of the product in the image based on the identified location of the barcode or other product label in the image; and
stitching, by the processor, each image of the second image data associated with the identified product to the three-dimensional mesh associated with the identified product based on the orientation of the product in each image of the second image data associated with the identified product.

### Claim 8

A system for three-dimensional product reconstruction using multiple images collected at POS workstations, comprising:
a barcode reader associated with a point of sale (POS) workstation configured to capture first image data associated with a plurality of products passing through a product scanning region of the POS workstation and analyze barcode data from the first image data to identify at least some of the plurality of products passing through the product scanning region of the POS workstation;
one or more color cameras associated with the POS workstation configured to, responsive to the barcode reader identifying a product, from the plurality of products, resulting in an identified product, capture second image data associated with the identified product;
a processor; and
a memory storing non-transitory, computer-readable instructions, that, when executed by the processor, cause the processor to generate a textured three-dimensional mesh reconstruction for the identified product based on the second image data associated with the identified product.

### Claim 9 (depends on 8)

The system of claim 8, wherein the instructions, that cause the processor to generate the textured three-dimensional mesh reconstruction associated with the identified product based on the second image data associated with the identified product, include instructions that cause the processor to:
for each image of the second image data associated with the identified product:
segment the identified product from the background, or any hands in the image;
analyze the segmented image of the identified product to determine a depth and a texture associated with each pixel of the identified product in the image; and
generate a three-dimensional point cloud associated with the segmented image of the identified product based on the depth and the texture associated with each pixel of the identified product; and
stitch the three-dimensional point clouds associated with each image of the second image data associated with the identified product together to generate a textured three-dimensional mesh reconstruction associated with the identified product.

### Claim 10 (depends on 9)

The method of claim 9, wherein the instructions, that cause the processor to stitch the three-dimensional point cloud associated with each image of the second image data associated with the identified product together to generate a textured three-dimensional mesh reconstruction associated with the product, include instructions that cause the processor to:
for each three-dimensional point cloud associated with the identified product:
identify a location of a barcode or other product label in the three-dimensional point cloud; and
determine an orientation of the product in the three-dimensional point cloud based on the identified location of the barcode or other product label in the three-dimensional point cloud; and
stitch the three-dimensional point clouds associated with each image of the second image data associated with the identified product together to generate the textured three-dimensional mesh reconstruction associated with the product, based on the determined orientation of the product in each three-dimensional point cloud.

### Claim 11 (depends on 8)

The method of claim 8, wherein the instructions, that cause the processor to generate the textured three-dimensional mesh reconstruction for the identified product based on the second image data associated with the identified product, include instructions that cause the processor to:
determine an indication of a shape associated with the identified product;
generate a three-dimensional mesh associated with the identified product based on the indication of the shape associated with the identified product; and
stitch the second image data associated with the identified product to the three-dimensional mesh associated with the identified product in order to generate a textured three-dimensional mesh associated with the identified product.

### Claim 12 (depends on 11)

The method of claim 11, wherein the instructions, that cause the processor to determine the indication of the shape associated with the identified product, include instructions that cause the processor to access a database storing an indication of a product shape associated with the identified product.

### Claim 13 (depends on 11)

The method of claim 11, wherein the instructions, that cause the processor to determine the indication of the shape associated with the product shown in the image, include instructions causing the processor to:
for at least one image of the second image data associated with the identified product:
segment the identified product from the background, or any hands in the image; and
analyze the segmented image of the identified product using a neural network to classify the shape associated with the identified product as one of a plurality of possible product shapes.

### Claim 14 (depends on 11)

The method of claim 11, wherein the instructions, that cause the processor to stitch the second image data associated with the identified product to the three-dimensional mesh associated with the identified product in order to generate the textured three-dimensional mesh associated with the identified product, include instructions that cause the processor to:
for each image of the second image data associated with the identified product:
segment the identified product from the background, or any hands in the image;
identify a location of a barcode or other product label in the image; and
determine an orientation of the product in the image based on the identified location of the barcode or other product label in the image; and
stitch each image of the second image data associated with the identified product to the three-dimensional mesh associated with the identified product based on the orientation of the product in each image of the second image data associated with the identified product.
