# US 2021/0241527 A1 · Point cloud generation method and system, and computer storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241527 A1
- Document ID: US20210241527A1
- Status: Published application
- Kind: A1
- Application number: 17/233,536
- Filed: 2021-04-18
- Published: 2021-08-05
- Earliest claimed: 2019-03-28
- Assignees: Sz Dji Technology Co., Ltd.
- Applicants: Sz Dji Technology Co., Ltd.
- Inventors: Chunmiao SUN; Jiabin LIANG
- Web page: https://patentgrep.com/patent/US20210241527A1

## CPC

- G06T 7/55: from multiple images
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/20224: Image subtraction
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A point cloud generation method and system, and a computer storage medium are provided. The method includes: initializing a spatial parameter of each pixel in a reference image; updating the spatial parameter of each pixel through propagation of adjacent pixels; comparing the spatial parameter of each pixel after the propagation with that before the propagation to determine whether a change occurs, if a change occurs, performing classification based on differences in a depth and a score after versus before the propagation, and changing the spatial parameter within a range; calculating the score of changed spatial parameter of each pixel, and updating spatial parameters of at least some pixels to spatial parameters whose scores reach a preset threshold range; determining, based on an updated spatial parameter of each pixel, a depth map corresponding to the reference image; and generating a dense point cloud image based on the depth map.

## Claims

### Claim 1

A point cloud generation method, comprising:
initializing a spatial parameter of each pixel in a reference image, wherein the reference image is an image in a two-dimensional image set obtained by photographing a target scene;
updating the spatial parameter of each pixel in the reference image through propagation of the spatial parameter of each pixel to adjacent pixels in the reference image to obtain an updated spatial parameter of the pixel in the reference image;
for each pixel in a reference image that a change of the spatial parameter occurs before and after the propagation:
determining a first difference between a depth value of the pixel before and after the propagation and a second difference between a score of the special parameter before and after the propagation;
performing classification based on the first difference and the second difference;
changing the spatial parameter of the pixel within a predetermined range based on the classification;
updating, based on the score of the updated spatial parameter of each pixel, spatial parameters of at least some pixels in the reference image to corresponding updated spatial parameters whose scores reach a preset threshold range;
determining, based on the updated spatial parameter of each pixel in the reference image, a depth map corresponding to the reference image; and
generating a dense point cloud image based on the depth map corresponding to the reference image.

### Claim 2 (depends on 1)

The point cloud generation method according to claim 1, wherein the spatial parameter of the pixel includes at least a depth value and a normal vector of a three-dimensional spatial point corresponding to the pixel.

### Claim 3 (depends on 1)

The point cloud generation method according to claim 1, wherein the preset threshold is in a positive correlation with the difference in the score.

### Claim 4 (depends on 1)

The point cloud generation method according to claim 1, wherein the preset threshold is in a negative correlation with the difference in the score.

### Claim 5 (depends on 1)

The point cloud generation method according to claim 1, wherein the score is obtained by:
projecting an image block around the pixel in the reference image to a neighboring image of the image set adjacent to the reference image; and
calculating a similarity score between the image block and a matched image block in the neighboring image,
wherein the image block is formed by a current pixel as a center pixel and a plurality of pixels spaced apart from the current pixel by at least one pixel selected in a current image block around the current pixel.

### Claim 6 (depends on 5)

The point cloud generation method according to claim 5, wherein the plurality of pixels are spaced apart from each other by at least one pixel.

### Claim 7 (depends on 5)

The point cloud generation method according to claim 5, wherein the calculating of the similarity score between the image block and the matched image block includes:
calculating a selection probability; and
calculating the similarity score by weighting a matching cost based on the selection probability, wherein
the selection probability is a probability of each pixel in the neighboring image appearing in the reference image.

### Claim 8 (depends on 1)

The point cloud generation method according to claim 1, wherein the predetermined range includes a first predetermined range and a second predetermined range;
an interval of the first predetermined range is larger than an interval of the second predetermined range; and
the changing of the spatial parameter of the pixel within the predetermined range includes at least one of:
randomly changing the spatial parameter of the pixel within the first predetermined range, or
fluctuating the spatial parameter of the pixel within the second predetermined range.

### Claim 9 (depends on 8)

The point cloud generation method according to claim 8, wherein the performing of the classification based on the first difference and the second difference and the changing of the spatial parameter of the pixel within the predetermined range based on the classification include:
determining that a current optimal estimate of a spatial parameter of a current pixel is obtained through the propagation of the spatial parameter of each pixel to the adjacent pixels; and
determine whether a difference between a current depth and a depth before the propagation exceeds a specified depth difference threshold.

### Claim 10 (depends on 9)

The point cloud generation method according to claim 9, further comprising:
determining that the difference between the current depth and the depth before the propagation exceeds the specified depth difference threshold; and
determining whether a difference between the score of the spatial parameter of the current pixel and the corresponding score before the propagation exceeds a specified score difference threshold.

### Claim 11 (depends on 10)

The point cloud generation method according to claim 10, further comprising:
determining that the difference between the score of the spatial parameter of the current pixel and the corresponding score before the propagation exceeds the specified score difference threshold and the score of the spatial parameter of the current pixel reaches the preset threshold range, and fluctuating the spatial parameter of the current pixel within the second predetermined range; or
determining that the difference between the score of the spatial parameter of the current pixel and the corresponding score before the propagation does not exceed the specified score difference threshold, and randomly changing the spatial parameter of the current pixel within the first predetermined range.

### Claim 12 (depends on 9)

The point cloud generation method according to claim 9, wherein the performing of the classification based on the first difference and the second difference and the changing of the spatial parameter of the pixel within the predetermined range based on the classification further include:
determining that the difference between the current depth and the depth before the propagation does not exceed the specified depth difference threshold; and
determining whether a difference between the score of the spatial parameter of the current pixel and the corresponding score before the propagation exceeds a specified score difference threshold;
upon determining that the difference between the score of the spatial parameter of the current pixel and the corresponding score before the propagation exceeds the specified score difference threshold and the score of the spatial parameter of the current pixel reaches the preset threshold range, fluctuating the spatial parameter of the current pixel within the second predetermined range, or
upon determining that the difference between the score of the spatial parameter of the current pixel and the corresponding score before the propagation does not exceed the specified score difference threshold, randomly changing the spatial parameter of the current pixel within the first predetermined range and fluctuating the spatial parameter of the current pixel within the second predetermined range.

### Claim 13 (depends on 8)

The point cloud generation method according to claim 8, wherein the performing of the classification based on the first difference and the second difference and the changing of the spatial parameter of the pixel within the predetermined range based on the classification further include:
determining that a current optimal estimate of a spatial parameter of a current pixel is not obtained through the propagation of adjacent pixels; and
determine whether a difference between a current depth and a depth before the propagation exceeds a specified depth difference threshold,
upon determining that the difference between the current depth and the depth before the propagation exceeds the specified depth difference threshold, determining whether a difference between the score of the spatial parameter of the current pixel and the corresponding score before the propagation exceeds a specified score difference threshold,
upon determining that the difference between the score of the spatial parameter of the current pixel and the corresponding score before the propagation exceeds the specified score difference threshold, randomly changing the spatial parameter of the current pixel within the first predetermined range, and fluctuating the spatial parameter of the current pixel within the second predetermined range, or
upon determining that the difference between the score of the spatial parameter of the current pixel and the corresponding score before the propagation does not exceed the specified score difference threshold, randomly changing the spatial parameter of the current pixel within the first predetermined range.

### Claim 14 (depends on 13)

The point cloud generation method according to claim 13, wherein the performing of the classification based on the first difference and the second difference and changing of the spatial parameter of the pixel within the predetermined range based on the classification further include:
upon determining that the difference between the current depth and the depth before the propagation does not exceed the specified depth difference threshold, determining whether the difference between the score of the spatial parameter of the current pixel and the corresponding score before the propagation exceeds the specified score difference threshold; and
upon determining that the difference between the score of the spatial parameter of the current pixel and the corresponding score before the propagation exceeds the specified score difference threshold, fluctuating the spatial parameter of the current pixel within the second predetermined range, or
upon determining that the difference between the score of the spatial parameter of the current pixel and the corresponding score before the propagation does not exceed the specified score difference threshold, randomly changing the spatial parameter of the current pixel within the first predetermined range, and fluctuating the spatial parameter of the current pixel within the second predetermined range.

### Claim 15 (depends on 8)

The point cloud generation method according to claim 8, wherein the first predetermined range includes a first predetermined range of depth and a first predetermined range of normal vector, and the randomly changing of the spatial parameter of the pixel within the first predetermined range includes:
keeping a depth of the pixel unchanged, and randomly changing a normal vector of the pixel within the first predetermined range of normal vector; and
keeping the normal vector of the pixel unchanged, and randomly changing the depth of the pixel within the first predetermined range of depth.

### Claim 16 (depends on 8)

The point cloud generation method according to claim 8, wherein the second predetermined range includes a second predetermined range of depth and a second predetermined range of normal vector, and the fluctuating of the spatial parameter of the pixel within the second predetermined range includes:
keeping a depth of the pixel unchanged, and fluctuating a normal vector thereof within the second predetermined range of normal vector; and
keeping the normal vector of the pixel unchanged, and fluctuating the depth of the pixel within the second predetermined range of depth.

### Claim 17 (depends on 1)

The point cloud generation method according to claim 1, wherein the initializing of the spatial parameter of the pixel in the reference image includes:
generating a sparse point cloud image based on the two-dimensional image set; and
initializing the spatial parameter of the pixel in the reference image based on the sparse point cloud image.

### Claim 18 (depends on 17)

The point cloud generation method according to claim 17, wherein the generating of the sparse point cloud image based on the two-dimensional image set includes:
generating the sparse point cloud image based on the two-dimensional image set by using a Structure from Motion method.

### Claim 19 (depends on 1)

The point cloud generation method according to claim 1, wherein a direction of the propagation of the spatial parameter of the pixel to adjacent pixels includes:
at least one of from left to right of the reference image, from right to left of the reference image, from top to bottom of the reference image, or from bottom to top of the reference image.

### Claim 20 (depends on 1)

The point cloud generation method according to claim 1, wherein the determining, based on the updated spatial parameter of each pixel in the reference image, of the depth map corresponding to the reference image includes:
upon the spatial parameter of each pixel in the reference image converging to a stable value, determining, based on the spatial parameter of each pixel in the reference image, the depth map corresponding to the reference image.
