# US 2024/0005591 A1 · Method and system for rendering panoramic video

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0005591 A1
- Document ID: US20240005591A1
- Status: Published application
- Kind: A1
- Application number: 18/038,407
- Filed: 2021-08-12
- Published: 2024-01-04
- Earliest claimed: 2020-11-25
- Assignees: Shanghai Bilibili Technology Co., Ltd.
- Applicants: Shanghai Bilibili Technology Co., Ltd.
- Inventors: Huan XU; Zhaoxin TAN; Hao LIU
- Web page: https://patentgrep.com/patent/US20240005591A1

## CPC

- G06T 15/205: {Image-based rendering}
- H04N 21/816: {involving special video data, e.g 3D video}
- G06T 15/04: Texture mapping
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- H04N 21/4424: {Monitoring of the internal components or processes of the client device, e.g. CPU or memory load, processing speed, timer, counter or percentage of the hard disk space used (error monitoring in general G06F11/30; arrangements for monitoring conditions of receiving stations in broadcast systems H04H60/32; diagnosis, testing or measuring for television receivers H04N17/04)}
- H04N 21/4621: {Controlling the complexity of the content stream or additional data, e.g. lowering the resolution or bit-rate of the video stream for a mobile client with a small screen (arrangements for using the results of monitoring on user's side in broadcast systems H04H60/65; flow control in packet networks H04L47/10)}
- H04N 21/4728: for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2219/2016: Rotation, translation, scaling
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

The present application discloses a method for rendering a panoramic video. The method includes obtaining a current frame of image from a video source, and generating texture map data, where the generating texture map data comprises determining a viewpoint region based on a field of view of a perspective camera, and rendering image pixels outside the viewpoint region at a lower resolution than rendering image pixels within the viewpoint region; mapping the current frame of image to a three-dimensional image based on a spherical rendering model the texture map data; and projecting the three-dimensional image onto a two-dimensional screen.

## Claims

### Claim 1

A method for rendering a panoramic video, the method comprising:
obtaining a current frame of image from a video source;
generating texture map data corresponding to the current frame of image, wherein the generating texture map data comprises determining a viewpoint region based on a field of view of a perspective camera, and rendering image pixels outside the viewpoint region at a lower resolution than rendering image pixels within the viewpoint region, and wherein the perspective camera is constructed at a center of a three-dimensional sphere of a spherical rendering model as a point for a user to view the panoramic video;
mapping the current frame of image to a three-dimensional image based on the spherical rendering model and the texture map data; and
projecting the three-dimensional image onto a two-dimensional screen.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the determining a viewpoint region based on a field of view of a perspective camera comprises:
obtaining a field-of-view range of the perspective camera and a preset field-of-view threshold; and
determining the viewpoint region by adding a region indicated by the preset field-of-view threshold surrounding a region indicated by the field-of-view range.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the rendering image pixels outside the viewpoint region at a lower resolution than rendering image pixels within the viewpoint region comprises:
setting a first resolution in preset resolution levels for playing the panoramic video;
rendering the image pixels within the viewpoint region at the first resolution to generate first texture map data;
determining a second resolution in the preset resolution levels, wherein the second resolution is one level lower than the first resolution; and
rendering the image pixels outside the viewpoint region at the second resolution to generate second texture map data.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the projecting the three-dimensional image onto a two-dimensional screen comprises:
monitoring an interactive operation corresponding to a predetermined interaction manner associated with a client computing device;
recording a two-dimensional offset generated by the interactive operation;
converting the two-dimensional offset into an offset angle of the three-dimensional image in the spherical rendering model; and
updating a projection matrix of the perspective camera based on the offset angle.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the predetermined interaction manner associated with a client computing device comprises:
when the client computing device is a web-based computing device, the predetermined interaction manner comprises an interactive operation performed via a mouse or a keyboard; and
when the client computing device is a mobile computing device, the predetermined interaction manner comprises an operation performed via a touchscreen or an operation performed via a device of controlling field-of-view mode.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the projecting the three-dimensional image onto a two-dimensional screen further comprises:
monitoring whether a size of a presentation region for playing the panoramic video has changed;
in determining that the size of the presentation region has changed, obtaining a changed size of the presentation region; and
adjusting a size of a rendering canvas and updating a projection matrix of the perspective camera based on the changed size.

### Claim 7 (depends on 6)

The method according to claim 6, wherein the adjusting a size of a rendering canvas comprises:
setting the rendering canvas to a same size as the changed size of the presentation region.

### Claim 8 (depends on 1)

The method according to claim 1, wherein after the obtaining a current frame of image from a video source, the method further comprises:
determining whether the current frame of image needs to be rendered based on a preset performance detection parameter; and
in response to determining that the current frame of image does not need to be rendered, skipping the current frame of image and continuing to obtain a next frame of image.

### Claim 9

(canceled)

### Claim 10

(canceled)

### Claim 11

(canceled)

### Claim 12

An electronic device, comprising: a memory, a processor, and computer-readable instructions stored on the memory and executable on the processor, wherein when the computer-readable instructions are executed by the processor, the processor implements operations comprising:
obtaining a current frame of image from a video source;
generating texture map data corresponding to the current frame of image, wherein the generating texture map data comprises determining a viewpoint region based on a field of view of a perspective camera, and rendering image pixels outside the viewpoint region at a lower resolution than rendering image pixels within the viewpoint region, and wherein the perspective camera is constructed at a center of a three-dimensional sphere of a spherical rendering model as a point for a user to view a panoramic video;
mapping the current frame of image to a three-dimensional image based on the spherical rendering model and the texture map data; and
projecting the three-dimensional image onto a two-dimensional screen.

### Claim 13

A non-transitory computer-readable storage medium, having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by a processor, the processor implements operations comprising:
obtaining a current frame of image from a video source;
generating texture map data corresponding to the current frame of image, wherein the generating texture map data comprises determining a viewpoint region based on a field of view of a perspective camera, and rendering image pixels outside the viewpoint region at a lower resolution than rendering image pixels within the viewpoint region, and wherein the perspective camera is constructed at a center of a three-dimensional sphere of a spherical rendering model as a point for a user to view the panoramic video;
mapping the current frame of image to a three-dimensional image based on the spherical rendering model and the texture map data; and
projecting the three-dimensional image onto a two-dimensional screen.

### Claim 14 (depends on 12)

The electronic device according to claim 12, wherein the determining a viewpoint region based on a field of view of a perspective camera comprises:
obtaining a field-of-view range of the perspective camera and a preset field-of-view threshold; and
determining the viewpoint region by adding a region indicated by the preset field-of-view threshold surrounding a region indicated by the field-of-view range.

### Claim 15 (depends on 12)

The electronic device according to claim 12, wherein the rendering image pixels outside the viewpoint region at a lower resolution than rendering image pixels within the viewpoint region comprises:
setting a first resolution in preset resolution levels for playing the panoramic video;
rendering the image pixels within the viewpoint region at the first resolution to generate first texture map data;
determining a second resolution in the preset resolution levels, wherein the second resolution is one level lower than the first resolution; and
rendering the image pixels outside the viewpoint region at the second resolution to generate second texture map data.

### Claim 16 (depends on 12)

The electronic device according to claim 12, wherein the projecting the three-dimensional image onto a two-dimensional screen comprises:
monitoring an interactive operation corresponding to a predetermined interaction manner associated with a client computing device;
recording a two-dimensional offset generated by the interactive operation;
converting the two-dimensional offset into an offset angle of the three-dimensional image in the spherical rendering model; and
updating a projection matrix of the perspective camera based on the offset angle.

### Claim 17 (depends on 16)

The electronic device according to claim 16, wherein the predetermined interaction manner associated with a client computing device comprises:
when the client computing device is a web-based computing device, the predetermined interaction manner comprises an interactive operation performed via a mouse or a keyboard; and
when the client computing device is a mobile computing device, the predetermined interaction manner comprises an operation performed via a touchscreen or an operation performed via a device of controlling field-of-view mode.

### Claim 18 (depends on 12)

The electronic device according to claim 12, wherein the projecting the three-dimensional image onto a two-dimensional screen further comprises:
monitoring whether a size of a presentation region for playing the panoramic video has changed;
in determining that the size of the presentation region has changed, obtaining a changed size of the presentation region; and
adjusting a size of a rendering canvas and updating a projection matrix of the perspective camera based on the changed size.

### Claim 19 (depends on 18)

The electronic device according to claim 18, wherein the adjusting a size of a rendering canvas comprises:
setting the rendering canvas to a same size as the changed size of the presentation region.

### Claim 20 (depends on 12)

The electronic device according to claim 12, wherein after the obtaining a current frame of image from a video source, the operations further comprise:
determining whether the current frame of image needs to be rendered based on a preset performance detection parameter; and
in response to determining that the current frame of image does not need to be rendered, skipping the current frame of image and continuing to obtain a next frame of image.

### Claim 21 (depends on 13)

The non-transitory computer-readable storage medium according to claim 13, wherein the determining a viewpoint region based on a field of view of a perspective camera comprises:
obtaining a field-of-view range of the perspective camera and a preset field-of-view threshold; and
determining the viewpoint region by adding a region indicated by the preset field-of-view threshold surrounding a region indicated by the field-of-view range.

### Claim 22 (depends on 13)

The non-transitory computer-readable storage medium according to claim 13, wherein the projecting the three-dimensional image onto a two-dimensional screen comprises:
monitoring an interactive operation corresponding to a predetermined interaction manner associated with a client computing device;
recording a two-dimensional offset generated by the interactive operation;
converting the two-dimensional offset into an offset angle of the three-dimensional image in the spherical rendering model; and
updating a projection matrix of the perspective camera based on the offset angle.

### Claim 23 (depends on 13)

The non-transitory computer-readable storage medium according to claim 13, wherein the projecting the three-dimensional image onto a two-dimensional screen further comprises:
monitoring whether a size of a presentation region for playing the panoramic video has changed;
in determining that the size of the presentation region has changed, obtaining a changed size of the presentation region; and
adjusting a size of a rendering canvas and updating a projection matrix of the perspective camera based on the changed size.
