# US 2021/0334533 A1 · Object identification method and object identification device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334533 A1
- Document ID: US20210334533A1
- Status: Published application
- Kind: A1
- Application number: 16/481,542
- Filed: 2019-01-02
- Published: 2021-10-28
- Earliest claimed: 2018-04-28
- Assignees: Boe Technology Group Co., Ltd.
- Applicants: Boe Technology Group Co., Ltd.
- Inventors: Tong LIU; Xiangyang LIU; Zhiguo ZHANG; Hui RAO
- Web page: https://patentgrep.com/patent/US20210334533A1

## CPC

- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/248: {involving reference images or patches}
- G06T 7/254: involving subtraction of images
- G06T 7/337: {involving reference images or patches}
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60)
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20224: Image subtraction
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An object identification method and an object identification device, the object identification method comprises: capturing color images and depth images that are aligned; performing motion detection on the captured color images and depth images to acquire a motion start frame and a motion end frame; comparing the motion end frame with the motion start frame in the color images to acquire a set of motion pixels; performing connected component analysis on the acquired set of motion pixels to obtain at least one connected component; and identifying an object in the at least one connected component.

## Claims

### Claim 1

An object identification method comprising:
capturing color images and depth images that are aligned;
performing motion detection on the captured color images and depth images to acquire a motion start frame and a motion end frame;
comparing the motion end frame with the motion start frame in the color images to acquire a set of motion pixels;
performing connected component analysis on the acquired set of motion pixels to obtain at least one connected component; and
identifying an object in the at least one connected component.

### Claim 2 (depends on 1)

The method of claim 1, further comprising dividing the at least one connected component into at least one object component based on depth information of the motion end frame in the depth images; and the step of identifying an object in the at least one connected component comprises identifying the object in the at least one object component.

### Claim 3 (depends on 2)

The method of claim 2, wherein the performing motion detection on the captured color images and depth images to acquire a motion start frame and a motion end frame comprises:
determining, for the captured color images, whether a number of pixels whose difference in pixel values between an i-th frame and an (i−1)-th frame is greater than a first threshold exceeds a second threshold, where i is a positive integer greater than 1;
determining, for the captured depth images, whether a number of pixels whose difference in pixel values between an i-th frame and an (i−1)-th frame is greater than a third threshold exceeds a fourth threshold;
determining the i-th frame to be a motion frame in response to at least one of the following conditions, otherwise determining the i-th frame to be a still frame:
(1) the number of pixels whose difference in pixel values between the i-th frame and the (i−1)-th frame of the color images is greater than the first threshold exceeds the second threshold; and
(2) the number of pixels whose difference in pixel values between the i-th frame and the (i−1)-th frame of the depth images is greater than the third threshold exceeds the fourth threshold; and
determining the motion start frame and the motion end frame in the color images and the depth images based on the motion frame and the still frame.

### Claim 4 (depends on 2)

The method of claim 2, wherein the performing motion detection on the captured color images and depth images to acquire the motion start frame and the motion end frame comprises:
determining, for the captured color images, a number X1 of pixels whose difference in pixel values between an i-th frame and an (i−1)-th frame is greater than a first threshold, where i is a positive integer greater than 1;
determining, for the captured depth images, a number X2 of pixels whose difference in pixel values between an i-th frame and an (i−1)-th frame is greater than a third threshold;
determining whether a sum of X1 and X2 is greater than a fifth threshold, and if so, determining the i-th frame to be a motion frame, otherwise determining the i-th frame to be a still frame; and
determining the motion start frame and the motion end frame in the color images and the depth images based on the motion frame and the still frame.

### Claim 5 (depends on 3)

The method of claim 3, wherein the determining the motion start frame and the motion end frame in the color images and the depth images based on the motion frame and the still frame comprises:
determining a previous frame of a detected first motion frame as the motion start frame, and determining a N-th consecutive still frame after the first motion frame as the motion end frame, where N is a preset positive integer.

### Claim 6 (depends on 1)

The method of claim 1, wherein the comparing the motion end frame with the motion start frame in the color images to acquire a set of motion pixels comprises:
performing a pixel-by-pixel comparison of the motion end frame and the motion start frame in the color images to acquire a set of pixels whose difference in pixel values is greater than a sixth threshold; and
defining the set of pixels as the set of motion pixels.

### Claim 7 (depends on 2)

The method of claim 2, wherein the dividing the at least one connected component into at least one object component based on depth information of the motion end frame in the depth images comprises:
determining a pixel having a depth value of zero in the motion end frame in the depth images; and
dividing the at least one connected component into at least one object component along the pixel having the depth value of zero.

### Claim 8 (depends on 2)

The method of claim 2, wherein the identifying an object in the at least one object component comprises identifying an object in the at least one object component based on a depth neural network.

### Claim 9 (depends on 2)

The method of claim 2, wherein the performing connected component analysis on the acquired set of pixels to obtain at least one connected component comprises:
performing connected component analysis on the acquired set of pixels to obtain an initial connected component; and
removing a connected component that does not meet the preset condition for area or aspect ratio, or that is located inside other connected components from the initial connected component to obtain the at least one connected component.

### Claim 10 (depends on 4)

The method of claim 4, wherein the value of N is dynamically set based on the capture frequency of the color image and the depth image that are aligned.

### Claim 11

An object identification method comprising:
capturing color images and depth images that are aligned;
acquiring a start frame and an end frame in the captured color image and depth images;
comparing the end frame with the start frame in the color images to acquire a set of motion pixels;
performing connected component analysis on the acquired set of motion pixels to obtain at least one connected component;
dividing the at least one connected component into at least one object component based on depth information of the end frame in the depth image; and
identifying an object in the at least one object component.

### Claim 12 (depends on 11)

The method of claim 11, wherein the comparing the end frame with the start frame in the color images to acquire the set of motion pixels comprises:
performing a pixel-by-pixel comparison of the end frame and the start frame in the color images to acquire a set of pixels whose difference in pixel values is greater than a sixth threshold; and
defining the set of pixels as the set of motion pixels.

### Claim 13 (depends on 12)

The method of claim 12, wherein the dividing the at least one connected component into at least one object component based on depth information of the end frame in the depth images comprises:
determining a pixel having a depth value of zero in the end frame in the depth images; and
dividing the at least one connected component into at least one object component along the pixel having the depth value of zero.

### Claim 14

An object identification device comprising:
a capturing member, configured to capture color images and depth images that are aligned;
a detecting member, configured to perform motion detection on the captured color images and the depth images to acquire a motion start frame and a motion end frame;
an acquiring member, configured to compare the motion end frame with the motion start frame in the color images to acquire a set of motion pixels;
an analyzing member, configured to perform connected component analysis on the acquired set of motion pixels to obtain at least one connected component;
a dividing member, configured to divide the at least one connected component into at least one object component based on depth information of the motion end frame in the depth images; and
an identifying member, configured to identify an object in the at least one object component.

### Claim 15 (depends on 14)

The object identification device of claim 14, wherein the capturing member comprises an RGBD camera.

### Claim 16 (depends on 14)

A shopping cart comprising the object identification device according to claim 14.

### Claim 17 (depends on 1)

A computer readable storage medium having stored thereon computer readable instructions that, when executed, perform the method of claim 1.

### Claim 18 (depends on 4)

The method of claim 4, wherein the determining the motion start frame and the motion end frame in the color images and the depth images based on the motion frame and the still frame comprises:
determining a previous frame of a detected first motion frame as the motion start frame,
and determining a N-th consecutive still frame after the first motion frame as the motion end frame, where N is a preset positive integer.

### Claim 19 (depends on 2)

The method of claim 2, wherein the comparing the motion end frame with the motion start frame in the color images to acquire a set of motion pixels comprises:
performing a pixel-by-pixel comparison of the motion end frame and the motion start frame in the color images to acquire a set of pixels whose difference in pixel values is greater than a sixth threshold; and
defining the set of pixels as the set of motion pixels.

### Claim 20 (depends on 3)

The method of claim 3, wherein the comparing the motion end frame with the motion start frame in the color images to acquire a set of motion pixels comprises:
performing a pixel-by-pixel comparison of the motion end frame and the motion start frame in the color images to acquire a set of pixels whose difference in pixel values is greater than a sixth threshold; and
defining the set of pixels as the set of motion pixels.
