# US 11,910,068 B2 · Panoramic render of 3D video

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,910,068 B2
- Document ID: US11910068B2
- Status: Granted
- Kind: B2
- Application number: 17/517,970
- Filed: 2021-11-03
- Published: 2022-02-24
- Granted: 2024-02-20
- Earliest claimed: 2019-06-17
- Assignees: Shanghai Bilibili Technology Co., Ltd.
- Applicants: Shanghai Bilibili Technology Co., Ltd.
- Inventors: Lin Fu; Hui Chen
- Web page: https://patentgrep.com/patent/US11910068B2

## CPC

- H04N 13/161: Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597)
- H04N 21/816: {involving special video data, e.g 3D video}
- G06F 3/01: Input arrangements or combined input and output arrangements for interaction between user and computer (G06F3/16 takes precedence)
- G06F 3/048: Interaction techniques based on graphical user interfaces [GUI]
- G06T 15/00: Three-dimensional [3D] image rendering
- H04N 13/139: Format conversion, e.g. of frame-rate or size
- H04N 19/597: specially adapted for multi-view video sequence encoding
- H04N 21/440245: {the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment}
- H04N 21/44218: {Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45)}
- H04N 21/4728: for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method is provided. The method includes: reading video information of a 3D video to be rendered; determining an image format of the 3D video according to the video information; generating a first spherical model and a second spherical model according to the image format; reading a frame image of the 3D video, where the frame image includes a first sub-image and a second sub-image; obtaining angle-of-view rotation data and image sealing data; rendering the frame image according to the angle-of-view rotation data and the image scaling data, where the first sub-image is rendered to the first spherical model to obtain a first spherical image, and the second sub-image is rendered to the second spherical model to obtain a second spherical image; and outputting images in a viewport area of the first spherical image and the second spherical image to a display device.

## Claims

### Claim 1

A method, comprising:
reading video information of the three-dimension (3D) video to be rendered;
determining an image format of the 3D video according to the video information;
generating a first spherical model and a second spherical model according to the image format;
reading a frame image of the 3D video, wherein the frame image comprises a first sub-image and a second sub-image;
obtaining angle-of-view rotation data and image scaling data;
rendering the frame image according to the angle-of-view rotation data and the image scaling data, wherein the first sub-image is rendered to the first spherical model to obtain a first spherical image, and the second sub-image is rendered to the second spherical model to obtain a second spherical image; and
outputting images in a viewport area of the first spherical image and the second spherical image to a display device,
wherein the video information comprises a height and a width of the frame image, and the determining an image format of the 3D video comprises:
calculating a ratio of the height to the width;
in response to determining the ratio is 1:1, determining that the image format of the 3D video is an up-down format; and
in response to determining the ratio is 1:4, determining that the image format of the 3D video is a left-right format.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the image format is one of an up-down format and a left-right format, and the generating a first spherical model and a second spherical model comprises:
in response to determining the image format is the up-down format, setting texture information of the frame image in a range of [0 to 1] in a first direction to correspond to [0 degrees to 360 degrees] of a sphere model, and texture information of the frame image in a range of [0 to 0.5] in a second direction to correspond to [90 degrees to 0 degrees to −90 degrees] of the sphere model, to obtain the first spherical model, and setting the texture information of the frame image in the range of [0 to 1] in the first direction to correspond to [0 degrees to 360 degrees] of the sphere model, and texture information of the frame image in a range of [0.5 to 1] in the second direction to correspond to [90 degrees to 0 degrees to −90 degrees] of the sphere model, to obtain the second spherical model; and
in response to determining the image format is the left-right format, setting texture information of the frame image in a range of [0 to 0.5] in a first direction to correspond to [0 degrees to 360 degrees] of a sphere model, and texture information of the frame image in a range of [0 to 1] in a second direction to correspond to [90 degrees to 0 degrees to −90 degrees] of the sphere model, to obtain the first spherical model, and setting the texture information of the frame image in the range of [0.5 to 1] in the first direction to correspond to [0 degrees to 360 degrees] of the sphere model, and texture information of the frame image in a range of [0 to 1] in the second direction to correspond to [90 degrees to 0 degrees to −90 degrees] of the sphere model, to obtain the second spherical model.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the angle-of-view rotation data comprises a pitch angle, a yaw angle, and a roll angle, and the image scaling data comprises a scale factor.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the obtaining angle-of-view rotation data and image scaling data comprises:
obtaining the angle-of-view rotation data output by a sensor in the display device; and
determining that the image scaling data is a default value.

### Claim 5 (depends on 3)

The method according to claim 3, wherein the obtaining angle-of-view rotation data and image scaling data comprises:
in response to a single-finger swipe operation on the display device, computing the angle-of-view rotation data; and
in response to a two-finger swipe operation on the display device, computing the image scaling data.

### Claim 6 (depends on 3)

The method according to claim 3, wherein the rendering the frame image according to the angle-of-view rotation data and the image scaling data comprises:
converting the pitch angle, the yaw angle, and the roll angle into a rotation matrix;
multiplying the rotation matrix by a model view transformation matrix in Open Graphics Library (OpenGL) to obtain a first matrix;
setting a field of view (FOV) parameter in a projection matrix in OpenGL to the scale factor to obtain a second matrix;
multiplying the first matrix by the second matrix to obtain a third matrix; and
rendering the frame image according to the third matrix.

### Claim 7 (depends on 3)

The method according to claim 3, wherein the rendering the frame image comprises:
converting the pitch angle, the yaw angle, and the roll angle into quaternions; and
rendering the frame image according to the quaternions and the scale factor.

### Claim 8 (depends on 3)

The method according to claim 3, wherein the rendering the frame image comprises:
inputting the pitch angle, the yaw angle, and the roll angle to a LookAt function in OpenGL to obtain a processing result, and
rendering, by a rendering function, the frame image according to the processing result and the scale factor.

### Claim 9

A computer device, comprising:
one or more processors; and
one or more memories storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for:
reading video information of a 3D video to be rendered;
determining an image format of the 3D video according to the video information;
generating a first spherical model and a second spherical model according to the image format;
reading a frame image of the 3D video, wherein the frame image comprises a first sub-image and a second sub-image;
obtaining angle-of-view rotation data and image scaling data;
rendering the frame image according to the angle-of-view rotation data and the image scaling data, wherein the first sub-image is rendered to the first spherical model to obtain a first spherical image, and the second sub-image is rendered to the second spherical model to obtain a second spherical image; and
outputting images in a viewport area of the first spherical image and the second spherical image to a display device,
wherein the one or more programs further comprise instructions for:
calculating a ratio of the height to the width;
in response to determining the ratio is 1:1, determining that the image format of the 3D video is an up-down format; and
in response to determining the ratio is 1:4, determining that the image format of the 3D video is a left-right format.

### Claim 10 (depends on 9)

The computer device according to claim 9, wherein the one or more programs further comprise instructions for:
in response to determining the image format is an up-down format, setting texture information of the frame image in a range of [0 to 1] in a first direction to correspond to [0 degrees to 360 degrees] of a sphere model, and texture information of the frame image in a range of [0 to 0.5] in a second direction to correspond to [90 degrees to 0 degrees to −90 degrees] of the sphere model, to obtain the first spherical model, and setting the texture information of the frame image in the range of [0 to 1] in the first direction to correspond to [0 degrees to 360 degrees] of the sphere model, and texture information of the frame image in a range of [0.5 to 1] in the second direction to correspond to [90 degrees to 0 degrees to −90 degrees] of the sphere model, to obtain the second spherical model; and
in response to determining the image format is a left-right format, setting texture information of the frame image in a range of [0 to 0.5] in a first direction to correspond to [0 degrees to 360 degrees] of a sphere model, and texture information of the frame image in a range of [0 to 1] in a second direction to correspond to [90 degrees to 0 degrees to −90 degrees] of the sphere model, to obtain the first spherical model, and setting the texture information of the frame image in the range of [0.5 to 1] in the first direction to correspond to [0 degrees to 360 degrees] of the sphere model, and texture information of the frame image in a range of [0 to 1] in the second direction to correspond to [90 degrees to 0 degrees to −90 degrees] of the sphere model, to obtain the second spherical model.

### Claim 11 (depends on 9)

The computer device according to claim 9, wherein the angle-of-view rotation data comprises a pitch angle, a yaw angle, and a roll angle, and the image scaling data comprises a scale factor.

### Claim 12 (depends on 11)

The computer device according to claim 11, wherein the one or more programs further comprise instructions for:
obtaining the angle-of-view rotation data output by a sensor in the display device; and
determining that the image scaling data is a default value.

### Claim 13 (depends on 11)

The computer device according to claim 11, wherein the one or more programs further comprise instructions for:
in response to a single-finger swipe operation on the display device, computing the angle-of-view rotation data; and
in response to a two-finger swipe operation on the display device, computing the image scaling data.

### Claim 14 (depends on 11)

The computer device according to claim 11, wherein the one or more programs further comprise instructions for:
converting the pitch angle, the yaw angle, and the roll angle into a rotation matrix;
multiplying the rotation matrix by a model view transformation matrix in Open Graphics Library (OpenGL) to obtain a first matrix;
setting a field of view (FOV) parameter in a projection matrix in OpenGL to the scale factor to obtain a second matrix;
multiplying the first matrix by the second matrix to obtain a third matrix; and
rendering the frame image according to the third matrix.

### Claim 15 (depends on 11)

The computer device according to claim 11, wherein the one or more programs further comprise instructions for:
converting the pitch angle, the yaw angle, and the roll angle into quaternions; and
rendering the frame image according to the quaternions and the scale factor.

### Claim 16

A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, upon executed by one or more processors of an electronic device, cause the electronic device to perform operations comprising:
reading video information of a 3D video to be rendered;
determining an image format of the 3D video according to the video information;
generating a first spherical model and a second spherical model according to the image format;
reading a frame image of the 3D video, wherein the frame image comprises a first sub-image and a second sub-image;
obtaining angle-of-view rotation data and image scaling data;
rendering the frame image according to the angle-of-view rotation data and the image scaling data, wherein the first sub-image is rendered to the first spherical model to obtain a first spherical image, and the second sub-image is rendered to the second spherical model to obtain a second spherical image; and
outputting images in a viewport area of the first spherical image and the second spherical image to a display device,
wherein the one or more programs further comprising instructions, upon executed by the one or more processors of the electronic device, cause the electronic device to perform operations comprising:
calculating a ratio of the height to the width;
in response to determining the ratio is 1:1, determining that the image format of the 3D video is an up-down format; and
in response to determining the ratio is 1:4, determining that the image format of the 3D video is a left-right format.

### Claim 17 (depends on 16)

The non-transitory computer-readable storage medium according to claim 16, wherein the one or more programs further comprising instructions, upon executed by the one or more processors of the electronic device, cause the electronic device to perform operations comprising:
in response to determining the image format is an up-down format, setting texture information of the frame image in a range of [0 to 1] in a first direction to correspond to [0 degrees to 360 degrees] of a sphere model, and texture information of the frame image in a range of [0 to 0.5] in a second direction to correspond to [90 degrees to 0 degrees to −90 degrees] of the sphere model, to obtain the first spherical model, and setting the texture information of the frame image in the range of [0 to 1] in the first direction to correspond to [0 degrees to 360 degrees] of the sphere model, and texture information of the frame image in a range of [0.5 to 1] in the second direction to correspond to [90 degrees to 0 degrees to −90 degrees] of the sphere model, to obtain the second spherical model; and
in response to determining the image format is a left-right format, setting texture information of the frame image in a range of [0 to 0.5] in a first direction to correspond to [0 degrees to 360 degrees] of a sphere model, and texture information of the frame image in a range of [0 to 1] in a second direction to correspond to [90 degrees to 0 degrees to −90 degrees] of the sphere model, to obtain the first spherical model, and setting the texture information of the frame image in the range of [0.5 to 1] in the first direction to correspond to [0 degrees to 360 degrees] of the sphere model, and texture information of the frame image in a range of [0 to 1] in the second direction to correspond to [90 degrees to 0 degrees to −90 degrees] of the sphere model, to obtain the second spherical model.