# US 2019/0318509 A1 · Visual quality enhancement of reconstructed point clouds via color smoothing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0318509 A1
- Document ID: US20190318509A1
- Status: Published application
- Kind: A1
- Application number: 16/242,255
- Filed: 2019-01-08
- Published: 2019-10-17
- Earliest claimed: 2018-04-12
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Madhukar Budagavi; Hossein Najaf-Zadeh; Tuan Ho
- Web page: https://patentgrep.com/patent/US20190318509A1

## CPC

- G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00)
- G06F 18/23: Clustering techniques
- G06T 5/70: Denoising; Smoothing
- G06T 7/90: Determination of colour characteristics
- G06T 9/001: {Model-based coding, e.g. wire frame}
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 10/764: using classification, e.g. of video objects
- G06V 20/653: {by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A decoding device, an encoding device and a method for point cloud decoding is disclosed. The method includes receiving a compressed bitstream. The method also includes decoding the compressed bitstream into 2D frames that represent a 3D point cloud, each of the 2D frames including a set of patches representing a cluster of pixels of the 3D point cloud. The method further includes identifying an occupancy map included in the compressed bitstream. The method additionally includes identifying valid boundary points located at a boundary of a patch of the set of patches based on the occupancy map and identifying valid neighboring points of each of the valid boundary points of the patch. The method also includes generating the 3D point cloud using the 2D frames based on the valid neighboring points and the valid boundary points.

## Claims

### Claim 1

A decoding device for point cloud decoding, the decoding device comprising:
a communication interface configured to receive a compressed bitstream; and
a processor operably coupled to the communication interface, wherein the processor is configured to:
decode the compressed bitstream into two-dimensional (2D) frames that represent a three-dimensional (3D) point cloud, each of the 2D frames including a set of patches representing a cluster of pixels of the 3D point cloud,
identify an occupancy map included in the compressed bitstream, the occupancy map indicating locations of points in the 2D frames that represent valid pixels of the 3D point cloud,
identify valid boundary points located at a boundary of a patch of the set of patches based on the occupancy map,
identify valid neighboring points of each of the valid boundary points of the patch, and
generate the 3D point cloud using the 2D frames based on the valid neighboring points and the valid boundary points.

### Claim 2 (depends on 1)

The decoding device of claim 1, wherein:
the 2D frames include a color frame representing color of the 3D point cloud; and
the processor is further configured to:
identify a flag within the compressed bitstream, wherein the flag indicates whether color smoothing is performed when generating the 3D point cloud; and
in response to identifying the flag, perform color smoothing on the valid neighboring points and the valid boundary points; and
to generate the 3D point cloud, the processor is configured to modify portions of the color frame by color smoothing the valid neighboring points and the valid boundary points.

### Claim 3 (depends on 2)

The decoding device of claim 2, wherein to perform color smoothing, the processor is configured to:
derive a local entropy value for the valid neighboring points and the valid boundary points;
compare the local entropy value to a first threshold;
when the local entropy value is smaller than the first threshold, calculate a centroid representing color values of the valid neighboring points and the valid boundary points;
compare color values of the centroid to color values of a first boundary point of the valid boundary points; and
when a difference between the color values of the centroid and the color values of the first boundary point is larger than a second threshold, replace the color values of the first boundary point with the color values of the centroid.

### Claim 4 (depends on 3)

The decoding device of claim 3, wherein to calculate the centroid, the processor is configured to:
identify red color values, green color values, and blue color values for each point of the valid neighboring points and the valid boundary points; and
average the red color values, the green color values, and the blue color values to generate the centroid.

### Claim 5 (depends on 3)

The decoding device of claim 3, wherein to derive the local entropy value is based on:
luminance of the valid neighboring points, or
color components of the valid neighboring points.

### Claim 6 (depends on 1)

The decoding device of claim 1, wherein to identify the valid boundary points, the processor is further configured to:
for each point in one of the 2D frames, determine whether a value assigned to a first point is set to a first value at a corresponding location of the first point in the occupancy map;
when the value assigned to the first point is set to the first value, identify that the first point is an invalid point;
when the value assigned to the first point is not set to the first value, identify whether a value assigned to a second point that is adjacent to the first point is set to the first value in the occupancy map; and
after identifying that the value assigned to the second point is set to the first value, classify the first point as one of the valid boundary points.

### Claim 7 (depends on 6)

The decoding device of claim 6, wherein:
the second point is one of eight points that surround the first point, and
the first value is set to zero in the occupancy map.

### Claim 8 (depends on 6)

The decoding device of claim 6, wherein to identify the valid neighboring points, the processor is further configured to:
after identifying that the value assigned to the second point is set to the first value, identify whether a third point that is adjacent to the first point is not set to the first value in the occupancy map;
when a value assigned to the third point is not set to the first value, identify whether the third point is one of the valid boundary points; and
after identifying that the third point is not one of the valid boundary points, classify the third point as one of the valid neighboring points.

### Claim 9 (depends on 8)

The decoding device of claim 8, wherein the processor is further configured to:
identify a set of metadata within the compressed bitstream, the set of metadata indicating a distance from the first point for color smoothing; and
continue to identify subsequent neighboring points that until the distance away from the first point is identified.

### Claim 10

An encoding device for point cloud encoding, the encoding device comprising:
a processor configured to:
generate, for a three-dimensional (3D) point cloud, two-dimensional (2D) frames, each of the 2D frames including a set of patches representing a cluster of pixels of the 3D point cloud,
generate a flag that signals when color smoothing is performed,
generate metadata indicating parameters associated with the color smoothing,
generate an occupancy map indicating locations of points in the 2D frames that represent valid pixels of the 3D point cloud, and
encode the 2D frames, the flag, the metadata, and the occupancy map to generate a compressed bitstream; and
a communication interface operably coupled to the processor, the communication interface configured to transmit the compressed bitstream.

### Claim 11 (depends on 10)

The encoding device of claim 10, wherein:
the metadata includes a first threshold and a second threshold, the first threshold indicating a local entropy level, and the second threshold indicating a color value for smoothing.

### Claim 12 (depends on 10)

The encoding device of claim 10, wherein:
the occupancy map includes multiple first values and multiple second values, each of the multiple first values corresponds to invalid points on the 2D frames, and each of the multiple second values corresponds to valid points on the 2D frames.

### Claim 13 (depends on 10)

The encoding device of claim 10, wherein:
the metadata is a first set of metadata; and
the processor is further configured to:
generate a second set of metadata indicating a quantity of valid neighboring points from a valid boundary point for color smoothing, and
encode the second set of metadata into the compressed bitstream.

### Claim 14

A method for point cloud decoding, the method comprising:
receiving a compressed bitstream;
decoding the compressed bitstream into two-dimensional (2D) frames that represent a three-dimensional (3D) point cloud, each of the 2D frames including a set of patches representing a cluster of pixels of the 3D point cloud;
identifying an occupancy map included in the compressed bitstream, the occupancy map indicating locations of points in the 2D frames that represent valid pixels of the 3D point cloud;
identifying valid boundary points located at a boundary of a patch of the set of patches based on the occupancy map;
identifying valid neighboring points of each of the valid boundary points of the patch; and
generating the 3D point cloud using the 2D frames based on the valid neighboring points and the valid boundary points.

### Claim 15 (depends on 14)

The method of claim 14, wherein:
the 2D frames include a color frame representing color of the 3D point cloud;
the method further comprises:
identifying a flag within the compressed bitstream, wherein the flag indicates whether color smoothing is performed when generating the 3D point cloud, and
in response to identifying the flag, performing color smoothing on the valid neighboring points and the valid boundary points; and
generating the 3D point cloud, comprises modifying portions of the color frame by color smoothing the valid neighboring points and the valid boundary points.

### Claim 16 (depends on 15)

The method of claim 15, wherein performing color smoothing comprises:
deriving a local entropy value for the valid neighboring points and the valid boundary points;
comparing the local entropy value to a first threshold;
when the local entropy value is smaller than the first threshold, calculating a centroid representing color values of the valid neighboring points and the valid boundary points;
comparing color values of the centroid to color values of a first boundary point of the valid boundary points; and
when a difference between the color values of the centroid and the color values of the first boundary point is larger than a second threshold, replacing the color values of the first boundary point with the color values of the centroid.

### Claim 17 (depends on 16)

The method of claim 16, wherein calculating the centroid comprises:
identifying red color values, green color values, and blue color values for each point of the valid neighboring points and the valid boundary points; and
averaging the red color values, the green color values, and the blue color values to generate the centroid.

### Claim 18 (depends on 16)

The method of claim 16, wherein deriving the local entropy value is based on:
luminance of the valid neighboring points, or
color components of the valid neighboring points.

### Claim 19 (depends on 14)

The method of claim 14, wherein identifying the valid boundary points comprises:
for each point in one of the 2D frames, determining whether a value assigned to a first point is set to a first value at a corresponding location of the first point in the occupancy map;
when the value assigned to the first point is set to the first value, identifying that the first point is an invalid point;
when the value assigned to the first point is not set to the first value, identifying whether a value assigned to a second point that is adjacent to the first point is set to the first value in the occupancy map; and
after identifying that the value assigned to the second point is set to the first value, classifying the first point as one of the valid boundary points.

### Claim 20 (depends on 19)

The method of claim 19, wherein to identify the valid neighboring points comprises:
after identifying that the value assigned to the second point is set to the first value, identifying whether a third point that is adjacent to the first point is not set to the first value in the occupancy map;
when a value assigned to the third point is not set to the first value, identifying whether the third point is one of the valid boundary points; and
after identifying that the third point is not one of the valid boundary points, classifying the third point as one of the valid neighboring points.