# US 12,223,710 B2 · Image cropping using depth information

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,223,710 B2
- Document ID: US12223710B2
- Status: Granted
- Kind: B2
- Application number: 17/455,907
- Filed: 2021-11-19
- Published: 2022-12-29
- Granted: 2025-02-11
- Earliest claimed: 2021-06-29
- Assignees: 7-Eleven, Inc.
- Applicants: 7-Eleven, Inc.
- Inventors: Sailesh Bharathwaaj Krishnamurthy; Sumedh Vilas Datar; Crystal Maung; Shantanu Yadunath Thakurdesai
- Web page: https://patentgrep.com/patent/US12223710B2

## CPC

- G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60)
- G06V 20/44: {Event detection}
- G01S 5/16: using electromagnetic waves other than radio waves
- G01S 7/4802: {using analysis of echo signal for target characterisation; Target signature; Target cross-section}
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 17/88: Lidar systems specially adapted for specific applications
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/24: Classification techniques
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 7/136: involving thresholding
- G06T 7/254: involving subtraction of images
- G06T 7/50: Depth or shape recovery
- G06T 7/73: using feature-based methods
- G06T 7/74: {involving reference images or patches}
- G06V 10/16: {using multiple overlapping images; Image stitching}
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 20/64: Three-dimensional [3D] objects
- G06V 20/653: {by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces}
- H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00)
- H04N 13/207: using a single two-dimensional [2D] image sensor
- H04N 13/25: using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- H04N 23/80: Camera processing pipelines; Components thereof
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G01G 21/22: Weigh pans or other weighing receptacles; Weighing platforms
- G01S 17/894: Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20056: Discrete and fast Fourier transform, [DFT, FFT]
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20132: Image cropping
- G06T 2207/20221: Image fusion; Image merging
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G01G: WEIGHING (sorting by weighing B07C5/16)

## Abstract

A device configured to capture a first image of an item on a platform using a camera and to determine a first number of pixels in the first image that corresponds with the item. The device is further configured to capture a first depth image of an item on the platform using a three-dimensional (3D) sensor and to determine a second number of pixels within the first depth image that corresponds with the item. The device is further configured to determine that the difference between the first number of pixels in the first image and the second number of pixels in the first depth image is less than the difference threshold value, to extract the plurality of pixels corresponding with the item in the first image from the first image to generate a second image, and to output the second image.

## Claims

### Claim 1

An item tracking system, comprising:
a camera configured to capture images of at least a portion of a platform; and
a three-dimensional (3D) sensor configured to capture depth images of at least a portion of the platform, wherein each pixel in the depth images comprises depth information identifying a distance between the 3D sensor and a surface in the depth image; and
a processor operably coupled to the camera and the 3D sensor, and configured to:
capture a first image of the item on the platform using the camera;
identify a plurality of pixels corresponding with the item in the first image;
determine a first number of pixels in the plurality of pixels corresponding with the item;
capture a first depth image of an item on the platform using the 3D sensor;
determine a second number of pixels within the first depth image corresponding with the item;
determine a difference between the first number of pixels in the plurality of pixels corresponding with the item in the first image and the second number of pixels in the first depth image;
compare the difference between the first number of pixels in the plurality of pixels corresponding with the item in the first image and the second number of pixels in the first depth image to a difference threshold value, wherein the difference threshold value identifies a maximum pixel difference;
determine that the difference between the first number of pixels in the plurality of pixels corresponding with the item in the first image and the second number of pixels in the first depth image is less than the difference threshold value;
extract the plurality of pixels corresponding with the item in the first image from the first image to generate a second image in response to the determination that the difference between the first number of pixels in the plurality of pixels corresponding with the item in the first image and the second number of pixels in the first depth image is less than the difference threshold value; and
output the second image.

### Claim 2 (depends on 1)

The system of claim 1, wherein:
outputting the second image comprises loading the second image into a machine learning model that is configured to output a first encoded vector based on features of the item that are present in the second image;
the first encoded vector comprises an array of numerical values; and
each numerical value describes an attribute of the item based on the second image.

### Claim 3 (depends on 1)

The system of claim 1, wherein:
the first depth image captures upward-facing surfaces of the item; and
the first image captures upward-facing surfaces of the item.

### Claim 4 (depends on 1)

The system of claim 1, wherein:
the first depth image captures side surfaces of the item; and
the first image captures side surfaces of the item.

### Claim 5 (depends on 1)

The system of claim 1, wherein the processor is further configured to:
capture a third image of the item on the platform using the camera;
identify a second plurality of pixels corresponding with the item in the third image;
determine a third number of pixels in the second plurality of pixels corresponding with the item in the third image;
determine a difference between the third number of pixels in the second plurality of pixels corresponding with the item in the third image and the second number of pixels in the first depth image;
compare the difference between the third number of pixels in the second plurality of pixels corresponding with the item in the third image and the second number of pixels in the first depth image to the difference threshold value;
determine the difference between the third number of pixels in the second plurality of pixels corresponding with the item in the third image and the second number of pixels in the first depth image is greater than the difference threshold value; and
discard the second plurality of pixels corresponding with the item in the third image.

### Claim 6 (depends on 1)

The system of claim 1, wherein the processor is further configured to determine an item type for the item based on physical attributes of the item that are present in the second image.

### Claim 7 (depends on 1)

The system of claim 1, further comprising a weight sensor; and
wherein the processor is further configured to determine a weight for the item using the weight sensor.

### Claim 8

An image cropping method, comprising:
capturing a first image of the item on the platform using a camera that is configured to capture images of at least a portion of a platform;
identifying a plurality of pixels corresponding with the item in the first image;
determining a first number of pixels in the plurality of pixels corresponding with the item;
capturing a first depth image of an item on the platform using a three-dimensional (3D) sensor that is configured to capture depth images of at least a portion of the platform, wherein each pixel in the depth images comprises depth information identifying a distance between the 3D sensor and a surface in the depth image;
determining a second number of pixels within the first depth image corresponding with the item;
determining a difference between the first number of pixels in the plurality of pixels corresponding with the item in the first image and the second number of pixels in the first depth image;
comparing the difference between the first number of pixels in the plurality of pixels corresponding with the item in the first image and the second number of pixels in the first depth image to a difference threshold value, wherein the difference threshold value identifies a maximum pixel difference;
determining that the difference between the first number of pixels in the plurality of pixels corresponding with the item in the first image and the second number of pixels in the first depth image is less than the difference threshold value;
extracting the plurality of pixels corresponding with the item in the first image from the first image to generate a second image in response to the determination that the difference between the first number of pixels in the plurality of pixels corresponding with the item in the first image and the second number of pixels in the first depth image is less than the difference threshold value; and
outputting the second image.

### Claim 9 (depends on 8)

The method of claim 8, wherein:
outputting the second image comprises loading the second image into a machine learning model that is configured to output a first encoded vector based on features of the item that are present in the second image;
the first encoded vector comprises an array of numerical values; and
each numerical value describes an attribute of the item based on the second image.

### Claim 10 (depends on 8)

The method of claim 8, wherein:
the first depth image captures upward-facing surfaces of the item; and
the first image captures upward-facing surfaces of the item.

### Claim 11 (depends on 8)

The method of claim 8, wherein:
the first depth image captures side surfaces of the item; and
the first image captures side surfaces of the item.

### Claim 12 (depends on 8)

The method of claim 8, further comprising:
capturing a third image of the item on the platform using the camera;
identifying a second plurality of pixels corresponding with the item in the third image;
determining a third number of pixels in the second plurality of pixels corresponding with the item in the third image;
determining a difference between the third number of pixels in the second plurality of pixels corresponding with the item in the third image and the second number of pixels in the first depth image;
comparing the difference between the third number of pixels in the second plurality of pixels corresponding with the item in the third image and the second number of pixels in the first depth image to the difference threshold value;
determining the difference between the third number of pixels in the second plurality of pixels corresponding with the item in the third image and the second number of pixels in the first depth image is greater than the difference threshold value; and
discarding the second plurality of pixels corresponding with the item in the third image.

### Claim 13 (depends on 8)

The method of claim 8, further comprising determining an item type for the item based on physical attributes of the item that are present in the second image.

### Claim 14 (depends on 8)

The method of claim 8, further comprising determining a weight for the item using a weight sensor.

### Claim 15

A computer program product comprising executable instructions stored in a non-transitory computer-readable medium that when executed by a processor causes the processor to:
capture a first image of the item on the platform using a camera that is configured to capture images of at least a portion of a platform;
identify a plurality of pixels corresponding with the item in the first image;
determine a first number of pixels in the plurality of pixels corresponding with the item;
capture a first depth image of an item on the platform using a three-dimensional (3D) sensor that is configured to capture depth images of at least a portion of the platform, wherein each pixel in the depth images comprises depth information identifying a distance between the 3D sensor and a surface in the depth image;
determine a second number of pixels within the first depth image corresponding with the item;
determine a difference between the first number of pixels in the plurality of pixels corresponding with the item in the first image and the second number of pixels in the first depth image;
compare the difference between the first number of pixels in the plurality of pixels corresponding with the item in the first image and the second number of pixels in the first depth image to a difference threshold value, wherein the difference threshold value identifies a maximum pixel difference;
determine that the difference between the first number of pixels in the plurality of pixels corresponding with the item in the first image and the second number of pixels in the first depth image is less than the difference threshold value;
extract the plurality of pixels corresponding with the item in the first image from the first image to generate a second image in response to the determination that the difference between the first number of pixels in the plurality of pixels corresponding with the item in the first image and the second number of pixels in the first depth image is less than the difference threshold value; and
output the second image.

### Claim 16 (depends on 15)

The computer program product of claim 15, wherein:
outputting the second image comprises loading the second image into a machine learning model that is configured to output a first encoded vector based on features of the item that are present in the second image;
the first encoded vector comprises an array of numerical values; and
each numerical value describes an attribute of the item based on the second image.

### Claim 17 (depends on 15)

The computer program product of claim 15, wherein:
the first depth image captures upward-facing surfaces of the item; and
the first image captures upward-facing surfaces of the item.

### Claim 18 (depends on 15)

The computer program product of claim 15, wherein:
the first depth image captures side surfaces of the item; and
the first image captures side surfaces of the item.

### Claim 19 (depends on 15)

The computer program product of claim 15, further comprising:
capturing a third image of the item on the platform using the camera;
identifying a second plurality of pixels corresponding with the item in the third image;
determining a third number of pixels in the second plurality of pixels corresponding with the item in the third image;
determining a difference between the third number of pixels in the second plurality of pixels corresponding with the item in the third image and the second number of pixels in the first depth image;
comparing the difference between the third number of pixels in the second plurality of pixels corresponding with the item in the third image and the second number of pixels in the first depth image to the difference threshold value;
determining the difference between the third number of pixels in the second plurality of pixels corresponding with the item in the third image and the second number of pixels in the first depth image is greater than the difference threshold value; and
discarding the second plurality of pixels corresponding with the item in the third image.

### Claim 20 (depends on 15)

The computer program product of claim 15, further comprising determining an item type for the item based on physical attributes of the item that are present in the second image.