# US 2021/0243420 A1 · Method and apparatus for processing 360-degree image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0243420 A1
- Document ID: US20210243420A1
- Status: Published application
- Kind: A1
- Application number: 17/238,864
- Filed: 2021-04-23
- Published: 2021-08-05
- Earliest claimed: 2016-02-12
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Eric YIP; Byeong-Doo CHOI; Jae-Yeon SONG
- Web page: https://patentgrep.com/patent/US20210243420A1

## CPC

- H04N 13/139: Format conversion, e.g. of frame-rate or size
- G06T 3/06: Topological mapping of higher dimensional structures onto lower dimensional surfaces
- G06T 3/12: Panospheric to cylindrical image transformations
- H04N 13/00: Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N 13/161: Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597)
- H04N 13/194: Transmission of image signals
- H04N 23/60: Control of cameras or camera modules
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A communication technique for merging, with an IoT technology, a 5G communication system for supporting a data transmission rate higher than that of a 4G system is provided. The communication technique can be applied to an intelligent service (for example, smart home, smart building, smart city, smart car or connected car, health care, digital education, retail business, and security and safety-related services, and the like) on the basis of a 5G communication technology and an IoT-related technology. A method for processing a 360-degree image is provided. The method includes determining a three-dimensional (3D) model for mapping a 360-degree image; determining a partition size for the 360-degree image; determining a rotational angle for each of the x, y, and z axes of the 360-degree image; determining an interpolation method to be applied when mapping the 360-degree image to a two-dimensional (2D) image; and converting the 360-degree image into the 2D image.

## Claims

### Claim 1

A method for processing 360-degree multi-view images, the method comprising:
identifying a two-dimensional (2D) image which is generated based on projecting a three-dimensional (3D) image to a planar space, and metadata related to the 2D image; and
transmitting the identified 2D image and the identified metadata,
wherein the 3D image is generated based on the 360-degree multi-view images, and
wherein the metadata comprise:
information indicating a 3D model for mapping the 2D image to the 3D model, among a plurality of 3D models, and
information indicating a rotating angle for each of three directions which is an x-axis, a y-axis and a z-axis, related to the 3D image.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
determining the 3D model, a partition size, the rotating angle for the each of three directions, and an interpolation method for the 3D image which minimizes distortion at a predetermined bitrate by comparing degrees of distortion for 2D images based on the plurality of the 3D models, a plurality of partition sizes, a plurality of rotating angles, and a plurality of interpolation methods,
wherein the metadata further comprises information indicating the determined partition size for the 3D image and the determined interpolation method for the 3D image.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
determining whether to apply a control grid interpolation method in which control points in partitioned regions of the 3D image are shifted in the direction of the x-axis and/or the y-axis by a predetermined distance to be varied for mapping each corner of the partitioned regions,
wherein the control points are corners of the partitioned regions.

### Claim 4 (depends on 2)

The method of claim 2, wherein the partition size for the 3D image is determined as a value to minimize an entire rate-distortion (RD) cost based on an amount of the metadata and the degree of distortion for the 2D image.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
converting non-background contents in the 3D image into first regions which is less distorted than second regions in the 2D image; and
converting background contents in the 3D image into the second regions in the 2D image.

### Claim 6 (depends on 1)

The method of claim 1, wherein the identified 2D image and the identified metadata are transmitted as a file.

### Claim 7

A method for processing 360-degree multi-view images, the method comprising:
receiving a two-dimensional (2D) image which is generated based on projecting a three-dimensional (3D) image to a planar space, and metadata related to the 2D image, wherein the 3D image is generated based on the 360-degree multi-view images; and
rendering the received 2D image based on the received metadata,
wherein the metadata comprise:
information indicating a 3D model for mapping the 2D image to the 3D model, among a plurality of 3D models, and
information indicating a rotating angle for each of three directions which is an x-axis, a y-axis and a z-axis, related to the 3D image.

### Claim 8 (depends on 7)

The method of claim 7, wherein the rendering of the received 2D image further comprises:
mapping the received 2D image to the 3D model based on the information indicating the 3D model, and
rotating the received 2D image based on the information indicating the rotating angle for the each of three directions.

### Claim 9 (depends on 7)

The method of claim 7, wherein the 2D image and the metadata are received as a file.

### Claim 10 (depends on 8)

The method of claim 8, further comprising:
applying a control grid interpolation method to the 2D image,
wherein the control grid interpolation method is a method in which control points in partitioned regions are shifted in a direction of the x-axis and/or the y-axis by a predetermined distance to be varied.

### Claim 11

A device for processing 360-degree multi-view images, the device comprising:
a transceiver, and
a processor coupled to the transceiver, wherein the processor is configured to:
identify a two-dimensional (2D) image which is generated based on projecting a three-dimensional (3D) image to a planar space, and metadata related to the 2D image, and
transmit the identified 2D image and the identified metadata,
wherein the 3D image is generated based on the 360-degree multi-view images,
wherein the metadata comprise:
information indicating a 3D model for mapping the 2D image to the 3D model, among a plurality of 3D models, and
information indicating a rotating angle for each of three directions which is an x-axis, a y-axis and a z-axis, related to the 3D image.

### Claim 12 (depends on 11)

The device of claim 11, wherein the processor is further configured to:
determine the 3D model, a partition size, the rotating angle for the each of three directions, and an interpolation method for the 3D image which minimizes distortion at a predetermined bitrate by comparing degrees of distortion for 2D images based on the plurality of the 3D models, a plurality of partition sizes, a plurality of rotating angles, and a plurality of interpolation methods,
wherein the metadata further comprises information indicating the determined partition size for the 3D image and the determined interpolation method for the 3D image.

### Claim 13 (depends on 11)

The device of claim 11, wherein the processor is further configured to:
determine whether to apply a control grid interpolation method in which control points in partitioned regions of the 3D image are shifted in the direction of the x-axis and/or the y-axis by a predetermined distance to be varied for mapping each corner of the partitioned regions,
wherein the control points are corners of the partitioned regions.

### Claim 14 (depends on 12)

The device of claim 12, wherein the partition size for the 3D image is determined as a value to minimize an entire rate-distortion (RD) cost based on an amount of the metadata and the degree of distortion for the 2D image.

### Claim 15 (depends on 11)

The device of claim 11, wherein the processor is further configured to:
convert non-background contents in the 3D image into first regions which is less distorted than second regions in the 2D image, and
convert background contents in the 3D image into the second regions in the 2D image.

### Claim 16 (depends on 11)

The device of claim 11, wherein the identified 2D image and the identified metadata are transmitted as a file.

### Claim 17

A device for processing 360-degree multi-view images, the device comprising:
a transceiver, and
a processor coupled to the transceiver, wherein the processor is configured to:
receive a two-dimensional (2D) image which is generated based on projecting a three-dimensional (3D) image to a planar space, and metadata related to the 2D image, wherein the 3D image is generated based on the 360-degree multi-view images, and
render the received 2D image based on the received metadata,
wherein the metadata comprise:
information indicating a 3D model for mapping the 2D image to the 3D model, among a plurality of 3D models, and
information indicating a rotating angle for each of three directions which is an x-axis, a y-axis and a z-axis, related to the 3D image.

### Claim 18 (depends on 17)

The device of claim 17, wherein the processor is further configured to:
map the received 2D image to the 3D model based on the information indicating the 3D model, and
rotate the received 2D image based on the information indicating the rotating angle for the each of three directions.

### Claim 19 (depends on 17)

The device of claim 17, wherein the 2D image and the metadata are received as a file.

### Claim 20 (depends on 18)

The device of claim 18, wherein the processor is further configured to:
apply a control grid interpolation method to the 2D image,
wherein the control grid interpolation method is a method in which control points in partitioned regions are shifted in a direction of the x-axis and/or the y-axis by a predetermined distance to be varied.
