# US 2020/0267369 A1 · Point cloud data communication system, point cloud data transmitting apparatus, and point cloud data transmission method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0267369 A1
- Document ID: US20200267369A1
- Status: Published application
- Kind: A1
- Application number: 16/417,693
- Filed: 2019-05-21
- Published: 2020-08-20
- Earliest claimed: 2019-02-19
- Assignees: Media Kobo, Inc.
- Applicants: Media Kobo, Inc.
- Inventors: Kazuo NAGASAWA; Kazuma OHASHI
- Web page: https://patentgrep.com/patent/US20200267369A1

## CPC

- H04N 13/194: Transmission of image signals
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
- H04N 13/167: Synchronising or controlling image signals
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- H04N 13/275: from three-dimensional [3D] object models, e.g. computer-generated stereoscopic image signals
- H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
- H04N 19/197: {including determination of the initial value of an encoding parameter (H04N19/56 takes precedence)}
- H04N 19/597: specially adapted for multi-view video sequence encoding
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A display request including designation related to a display state of a stereoscopic shape in a three-dimensional space is transmitted from a receiving apparatus 200 to a transmitting apparatus 100, and the transmitting apparatus 100 extracts point cloud data of a part corresponding to the display request from among the point cloud data stored in a point cloud data storage unit 10 in response to the display request and transmits the extracted point cloud data to the receiving apparatus 200. Accordingly, only the point cloud data of the part which is necessary for the display state of the stereoscopic shape which is actually displayed in the three-dimensional space by the receiving apparatus 200 is transmitted, and thus it is possible to transmit only the point cloud data necessary for display within a practically acceptable short time on a communication network 300 even though the entire data amount of the point cloud data stored in the point cloud data storage unit 10 is huge.

## Claims

### Claim 1

A point cloud data communication system, comprising a transmitting apparatus; and
a receiving apparatus,
wherein point cloud data is transmitted from the transmitting apparatus to the receiving apparatus via a communication network,
the transmitting apparatus includes
a point cloud data storage unit that stores point cloud data indicating a stereoscopic shape in a three-dimensional space,
a display request receiving unit that receives a display request transmitted from the receiving apparatus,
a point cloud data extracting unit that extracts point cloud data of a part corresponding to the display request received by the display request receiving unit from among the point cloud data stored in the point cloud data storage unit, and
a point cloud data transmitting unit that transmits the point cloud data extracted by the point cloud data extracting unit to the receiving apparatus, and
the receiving apparatus includes
a display request transmitting unit that transmits the display request including designation related to a display state of the stereographic shape in the three-dimensional space to the transmitting apparatus, and
a point cloud data receiving unit that receives the point cloud data transmitted from the transmitting apparatus in response to the display request transmitted by the display request transmitting unit.

### Claim 2 (depends on 1)

The point cloud data communication system according to claim 1, wherein the transmitting apparatus further includes a thinning-out unit that decides a part with a low priority in accordance with the display state of the stereographic shape specified on the basis of the designation included in the display request, and extracts point cloud data of the part with the low priority from among the point cloud data extracted by the point cloud data extracting unit, and
the point cloud data transmitting unit transmits the point cloud data remaining after thinning-out by the thinning-out unit among the point cloud data extracted by the point cloud data extracting unit to the receiving apparatus.

### Claim 3 (depends on 1)

The point cloud data communication system according to claim 1, wherein at least one of a viewpoint and a direction of a line of sight from the viewpoint is variable in the three-dimensional space,
the display request transmitting unit transmits the display request including at least one of the viewpoint and the direction of the line of sight designated by a user as the designation related to the display state of the stereoscopic shape to the transmitting apparatus, and
the point cloud data extracting unit extracts point cloud data of a part to which a priority is given when the stereoscopic shape is displayed in the three-dimensional space in accordance with at least one of the viewpoint and the direction of the line of sight included in the display request from among the point cloud data stored in the point cloud data storage unit.

### Claim 4 (depends on 3)

The point cloud data communication system according to claim 3, wherein the point cloud data extracting unit extracts point cloud data of a part to which a priority is given when the stereoscopic shape is displayed in the three-dimensional space in a field of view of a predetermined range specified on the basis of at least one of the viewpoint and the direction of the line of sight included in the display request from among the point cloud data stored in the point cloud data storage unit.

### Claim 5 (depends on 1)

The point cloud data communication system according to claim 1, wherein the point cloud data extracting unit extracts point cloud data of a part having a difference from a part at the time of previous extraction among the stereoscopic shapes displayed in the display state corresponding to the designation included in the display request from among the point cloud data stored in the point cloud data storage unit.

### Claim 6 (depends on 2)

The point cloud data communication system according to claim 2, wherein at least one of a viewpoint and a direction of a line of sight from the viewpoint is variable in the three-dimensional space,
the display request transmitting unit transmits the display request including at least one of the viewpoint and the direction of the line of sight designated by a user as the designation related to the display state of the stereoscopic shape to the transmitting apparatus,
the point cloud data extracting unit extracts point cloud data of a part to which a priority is given when the stereoscopic shape is displayed in the three-dimensional space in accordance with at least one of the viewpoint and the direction of the line of sight included in the display request from among the point cloud data stored in the point cloud data storage unit, and
the thinning-out unit decreases, when a part is thinned out from among the point cloud data extracted by the point cloud data extracting unit, a priority of a part related to a stereoscopic shape having a long distance from the viewpoint in accordance with a display state related to a perspective of the stereoscopic shape decided on the basis of the distance from the viewpoint, and increases a thinning-out amount of the point cloud data.

### Claim 7 (depends on 2)

The point cloud data communication system according to claim 2, wherein the thinning-out unit thins out point cloud data of a part having no difference from a part at the time of previous extraction among the stereoscopic shapes displayed in the display state corresponding to the designation included in the display request from among the point cloud data stored in the point cloud data storage unit.

### Claim 8 (depends on 3)

The point cloud data communication system according to claim 3, wherein the point cloud data stored in the point cloud data storage unit is point cloud data indicating a stereoscopic image in which the stereoscopic shapes included in a plurality of captured images are synthesized into one, the stereoscopic image being generated by setting a plurality of projection planes in accordance with a relative angular relation of a plurality of cameras installed to photograph a photographing target area from a plurality of angles, calculating depth information indicating distances of the stereoscopic shapes included in the plurality of captured images captured by the plurality of cameras from the cameras for each pixel, and projecting values of pixels of the stereoscopic shapes included in the plurality of captured images in directions of the projection planes corresponding to the captured images in accordance with the depth information, and
the point cloud data extracting unit decides a camera of a direction to which a priority is given in accordance with the viewpoint on the basis of a relation between the viewpoint and positions of the plurality of cameras, and extracts the point cloud data generated from the values of the pixels included in a captured image by the decided camera from among the point cloud data stored in the point cloud data storage unit as point cloud data of a part to which a priority is given when the stereographic shape is displayed in the three-dimensional space.

### Claim 9 (depends on 6)

The point cloud data communication system according to claim 6, wherein the point cloud data stored in the point cloud data storage unit is point cloud data indicating a stereoscopic image in which the stereoscopic shapes included in a plurality of captured images are synthesized into one, the stereoscopic image being generated by setting a plurality of projection planes in accordance with a relative angular relation of a plurality of cameras installed to photograph a photographing target area from a plurality of angles, calculating depth information indicating distances of the stereoscopic shapes included in the plurality of captured images captured by the plurality of cameras from the cameras for each pixel, and projecting values of pixels of the stereoscopic shapes included in the plurality of captured images in directions of the projection planes corresponding to the captured images in accordance with the depth information, and
the thinning-out unit calculates, when a part is thinned out from among the point cloud data extracted by the point cloud data extracting unit, a distance from the viewpoint to the stereoscopic shape on the basis of position information of each point projected in accordance with the depth information, and increases a thinning-out amount of point cloud data of a part related to a stereoscopic shape having a long distance from the viewpoint.

### Claim 10 (depends on 8)

The point cloud data communication system according to claim 8, wherein, instead of the point cloud data transmitting unit, the transmitting apparatus includes
an image data generating unit that generates a plurality of pieces of two-dimensional image data by projecting the point cloud data extracted by the point cloud data extracting unit onto a plurality of predetermined two-dimensional planes on the basis of the depth information,
a compression unit that performs image compression on the plurality of pieces of two-dimensional image data generated by the above image data generating unit, and
a transmitting unit that transmits the plurality of two-dimensional image data compressed by the compressing unit and the depth information to the receiving apparatus, and
instead of the point cloud data receiving unit, the receiving apparatus includes
a receiving unit that receives the plurality of pieces of two-dimensional image data and the depth information transmitted from the transmitting apparatus in response to the display request transmitted by the display request transmitting unit, and
a point cloud data generating unit that generates the point cloud data indicating a stereoscopic image in which the stereoscopic shapes included in a plurality of pieces of two-dimensional image data are synthesized into one by projecting the values of the pixels of the stereoscopic shapes included in the plurality of pieces of two-dimensional image data received by the receiving unit in directions of a plurality of projection planes in accordance with the depth information received by the receiving unit.

### Claim 11 (depends on 9)

The point cloud data communication system according to claim 9, wherein, instead of the point cloud data transmitting unit, the transmitting apparatus includes
an image data generating unit that generates a plurality of pieces of two-dimensional image data by projecting the point cloud data remaining after thinning-out by the thinning-out unit onto a plurality of predetermined two-dimensional planes on the basis of the depth information,
a compression unit that performs image compression on the plurality of pieces of two-dimensional image data generated by the image data generating unit, and
a transmitting unit that transmits the plurality of pieces of two-dimensional image data compressed by the compressing unit and the depth information to the receiving apparatus, and
instead of the point cloud data receiving unit, the receiving apparatus includes
a receiving unit that receives the plurality of pieces of two-dimensional image data and the depth information transmitted from the transmitting apparatus in response to the display request transmitted by the display request transmitting unit, and
a point cloud data generating unit that generates the point cloud data indicating a stereoscopic image in which the stereoscopic shapes included in a plurality of pieces of two-dimensional image data are synthesized into one by projecting the values of the pixels of the stereoscopic shapes included in the plurality of pieces of two-dimensional image data received by the receiving unit in directions of a plurality of projection planes in accordance with the depth information received by the receiving unit.

### Claim 12

A point cloud data transmitting apparatus that transmits point cloud data to a receiving apparatus via a communication network, comprising:
a point cloud data storage unit that stores point cloud data indicating a stereoscopic shape in a three-dimensional space;
a display request receiving unit that receives a display request including designation related to a display state of the stereoscopic shape in the three-dimensional space from the receiving apparatus;
a point cloud data extracting unit that extracts point cloud data of a part corresponding to the display request received by the display request receiving unit from among the point cloud data stored in the point cloud data storage unit; and
a point cloud data transmitting unit that transmits the point cloud data extracted by the point cloud data extracting unit to the receiving apparatus.

### Claim 13

A method of transmitting point cloud data indicating a stereoscopic shape in a three-dimensional space from a transmitting apparatus including a point cloud data storage unit that stores the point cloud data to a receiving apparatus via a communication network, comprising:
a step of receiving, by a display request receiving unit of the transmitting apparatus, a display request including designation related to a display state of the stereoscopic shape in the three-dimensional space from the receiving apparatus;
a step of extracting, by a point cloud data extracting unit of the transmitting apparatus, point cloud data of a part corresponding to the display request received by the display request receiving unit from among the point cloud data stored in the point cloud data storage unit; and
a step of transmitting, by the point cloud data transmitting unit of the transmitting apparatus, the point cloud data extracted by the point cloud data extracting unit to the receiving apparatus.
