# US 10,504,276 B2 · Re-utilization of render assets for video compression

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,504,276 B2
- Document ID: US10504276B2
- Status: Granted
- Kind: B2
- Application number: 16/161,327
- Filed: 2018-10-16
- Published: 2019-02-14
- Granted: 2019-12-10
- Earliest claimed: 2008-06-16
- Applicants: Julian Michael Urbach
- Inventors: Julian Michael Urbach
- Web page: https://patentgrep.com/patent/US10504276B2

## CPC

- G06T 15/08: Volume rendering
- G06T 15/005: {General purpose rendering architectures}
- H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence)
- H04N 19/597: specially adapted for multi-view video sequence encoding
- G06T 2200/16: involving adaptation to the client's capabilities
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Methods, apparatuses and systems directed to using viewport state data objects (VSDO) to render a series of video frames according to render instructions to achieve video compression. In a particular implementation, the video compression format exposes the VSDO and render instructions to a video render client, allowing the video render client to finish rendering a sequence of video frames from different spatial locations and view transform parameters. Implementations of this video compression format further allow any arbitrary client with enough graphics processing power and bandwidth to retransmit novel, completely rendered viewports to thin clients without taxing the root remote render devices.

## Claims

### Claim 1

A method comprising:
receiving, by a processor at a client host, a viewport state data object (VSDO) comprising a cubic map associated with a scene, the VSDO being generated based on at least a common reference point and for rendering of a plurality of frames associated with the scene;
determining, by the processor, that reference positions from which the plurality of frames are to be rendered are the same as the common reference point;
centering, by the processor, the cubic map at the common reference point;
receiving, by the processor, render instructions comprising at least respective transform parameters for the plurality of video frames to be rendered;
mapping, by the processor, the VSDO to a pixel shader as input for the rendering; and
rendering, by the processor, via the pixel shader, the plurality of video frames of the scene according to the render instructions.

### Claim 2 (depends on 1)

The method of claim 1, wherein rendering the plurality of video frames via the pixel shader comprises rendering the VSDO as a panoramic environment by sampling the VSDO using a view vector from the common reference point.

### Claim 3 (depends on 1)

The method of claim 1, wherein the mapping is to a pixel shader that is a GPU resource that renders pixels in a memory map in a parallel processing operation.

### Claim 4 (depends on 3)

The method of claim 3, further comprising mapping the pixel to a plurality of VSDOs.

### Claim 5 (depends on 1)

The method of claim 1, further comprising generating the VSDO on-the-fly or storing it for subsequent transmission in response to a request for a video.

### Claim 6 (depends on 1)

The method of claim 1, wherein the render instructions further comprise respective frame identifiers of the plurality of frames to be rendered, a VSDO identifier, and locations of the VSDO relative to other VSDOs.

### Claim 7 (depends on 1)

The method of claim 1, wherein the rendering of the cubic map comprises rendering the cubic map as layers that are rasterized, each of the layers comprising data associated with depth information of the scene from the common reference point.

### Claim 8 (depends on 1)

The method of claim 1, wherein the cubic map comprises six texture maps stitched into a cube.

### Claim 9 (depends on 8)

The method of claim 8, wherein the VSDO being generated based on at least the common reference point comprises generating at least one of the six texture maps by rendering the scene from the common reference point according to a respective view.

### Claim 10 (depends on 7)

The method of claim 7, wherein rendering the plurality of video frames via the pixel shader comprises computing a distance between a spatial location of one of the plurality of video frames and an object defined in the layers of the cubic map along a set of rays extending from the spatial location of the one of the plurality of video frames.

### Claim 11 (depends on 1)

The method of claim 1, wherein rendering, via the pixel shader the plurality of video frames comprises generating pixel values for at least one of the plurality of video frames.

### Claim 12 (depends on 1)

The method of claim 1, wherein receiving, at the client host, the VSDO comprises:
receiving, by the processor, one or more compressed files containing compressed versions of the VSDO; and
decompressing and constructing, by the processor, the VSDO from the compressed versions.

### Claim 13 (depends on 1)

The method of claim 1, further comprising:
rasterizing, by the processor, the rendered plurality of video frames from at least pre-determined spatial and view transform parameters using a fragment operation.

### Claim 14 (depends on 13)

The method of claim 13, wherein the spatial and view transform parameters are pre-determined according to the render instructions.

### Claim 15 (depends on 13)

The method of claim 13, wherein the spatial and view transform parameters are the pixel shader constants when performing the rendering on a GPU.

### Claim 16 (depends on 13)

The method of claim 13, wherein the fragment operation comprises, for each pixel of one of the plurality of video frames, performing a ray traced hit test on a plurality of VSDOs loaded in to a GPU for the one of the plurality of video frames by setting a spatial position as an origin of a ray and setting a direction vector of the ray to match the view transform parameters.

### Claim 17 (depends on 16)

The method of claim 16, wherein the ray hit test algorithm returns hit point and fragment layer information of layers of the cubic map of the plurality of VSDOs.

### Claim 18 (depends on 17)

The method of claim 17, wherein the fragment layer information comprises depth information of at least one of the plurality of VSDOs for the scene's static elements.

### Claim 19

A system, comprising:
a memory comprising instructions executable by one or more processors; and
one or more processors coupled to the memory and operable to execute the instructions, the memory comprising instructions for:
receiving, at a client host, a viewport state data object (VSDO) comprising a cubic map associated with a scene, the VSDO being generated based on at least a common reference point and for rendering of a plurality of frames associate with the scene;
determining that reference positions from which the plurality of frames are to be rendered are the same as the common reference point;
centering the cubic map at the common reference point;
receiving render instructions comprising at least respective transform parameters for the plurality of video frames to be rendered;
mapping the VSDO to a pixel shader as input for the rendering; and
rendering, via the pixel shader, the plurality of video frames of the scene according to the render instructions.

### Claim 20

One or more computer-readable storage media embodying software operable when executed by a computer system to:
receive, at a client host, a viewport state data object (VSDO) comprising a cubic map associated with a scene, the VSDO being generated based on at least a common reference point and for rendering of a plurality of frames associate with the scene;
determine that reference positions from which the plurality of frames are to be rendered are the same as the common reference point;
center the cubic map at the common reference point;
receive render instructions comprising at least respective transform parameters for the plurality of video frames to be rendered;
map the VSDO to a pixel shader as input for the rendering; and
render, via the pixel shader, the plurality of video frames of the scene according to the render instructions.