# US 2018/0322691 A1 · Real-time rendering with compressed animated light fields

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0322691 A1
- Document ID: US20180322691A1
- Status: Published application
- Kind: A1
- Application number: 15/971,950
- Filed: 2018-05-04
- Published: 2018-11-08
- Earliest claimed: 2017-05-05
- Assignees: Disney Enterprises, Inc.
- Applicants: Disney Enterprises, Inc.
- Inventors: Kenneth J. MITCHELL; Charalampos KONIARIS; Malgorzata E. KOSEK; David A. SINCLAIR
- Web page: https://patentgrep.com/patent/US20180322691A1

## CPC

- G06T 15/80: Shading
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06T 7/586: from multiple light sources, e.g. photometric stereo
- G06T 15/005: {General purpose rendering architectures}
- G06T 15/06: Ray-tracing
- G06T 15/08: Volume rendering
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems, methods, and articles of manufacture for real-time rendering using compressed animated light fields are disclosed. One embodiment provides a pipeline, from offline rendering of an animated scene from sparse optimized viewpoints to real-time rendering of the scene with freedom of movement, that includes three stages: offline preparation and rendering, stream compression, and real-time decompression and reconstruction. During offline rendering, optimal placements for cameras in the scene are determined, and color and depth images are rendered using such cameras. Color and depth data is then compressed using an integrated spatial and temporal scheme permitting high performance on graphics processing units for virtual reality applications. The compressed content may be decoded and reconstructed in real-time by selecting, using heuristics, cameras that provide useful data for a viewer, selecting grid cells from those cameras that are visible to the viewer, and using a ray marching technique to reconstruct the scene.

## Claims

### Claim 1

A computer-implemented method, comprising:
determining placements for a plurality of virtual cameras in a virtual scene, wherein the placements for the plurality of virtual cameras provide full motion light field visibility;
rendering the virtual scene using the virtual cameras at the determined placements, wherein the rendering produces color data and depth data;
compressing the color data and the depth data;
decompressing at least a portion of the compressed data; and
reconstructing one or more video frames using the decompressed portion of the compressed data.

### Claim 2 (depends on 1)

The method of claim 1, wherein the placements for the plurality of virtual cameras are determined based, at least in part, on an objective function which includes a term that penalizes camera redundancy.

### Claim 3 (depends on 2)

The method of claim 2, wherein:
the virtual scene includes a specular lighting environment; and
the term in the objective function that penalizes camera redundancy includes an angular weight term.

### Claim 4 (depends on 1)

The method of claim 1, wherein compressing the color data includes temporally compressing the color data, the temporal compression of the color data comprising:
dividing each of a plurality of rendered video frames into a plurality of grid cells; and
for each grid cell of the plurality of grid cells:
determining, using an optimization process, a smallest selection of keyframes of the grid cell that can be used to derive other frames of the grid cell, and
storing the determined keyframes and an array including parameter values for each of the other frames of the grid cell, wherein each of the parameter values indicates, for a corresponding one of the other frames, interpolations of closest keyframes to the one of the other frames forward and backward in time.

### Claim 5 (depends on 1)

The method of claim 1, wherein compressing the depth data includes temporally compressing the depth data, the temporal compression of the depth data comprising:
determining one or more keyframes from rendered depth frames; and
storing depth frames from the rendered depth frames that are not the one or more keyframes as P-frames that each encode differences to a last keyframe.

### Claim 6 (depends on 5)

The method of claim 5, wherein the differences to the last keyframe are encoded using a set of axis-aligned bounding boxes (AABBs) in a leaf level of an AABB hierarchy.

### Claim 7 (depends on 1)

The method of claim 1, wherein decompressing at least the portion of the compressed data includes:
selecting a plurality of the virtual cameras using one or more heuristics;
selecting grid cells from the selected cameras that are visible to a viewer; and
decompressing the selected grid cells.

### Claim 8 (depends on 7)

The method of claim 7, wherein the one or more heuristics include at least one of:
a distance-based heuristic which gives higher priority to viewpoints with closer distances to an eye location;
an angle-based heuristic which gives higher priority to viewpoints with smaller angles to the eye location;
an angle-based culling heuristic which culls any viewpoint forming an angle with another higher priority viewpoint that is smaller than a given threshold; and
a performance-based culling heuristic which culls viewpoints whose runtime performance is below a predefined threshold.

### Claim 9 (depends on 7)

The method of claim 7, wherein selecting the grid cells from the selected virtual cameras that are visible to the viewer includes:
determining whether a respective frustum corresponding to each grid cell of the grid cells and obtained by cutting off a volume of a pyramid for the grid cell before and after minimum and maximum depth values, respectively, intersects with a frustum of a viewing virtual camera; and
selecting cells associated with intersections between the frusta corresponding to the grid cells and the frustum of the viewing virtual camera.

### Claim 10 (depends on 7)

The method of claim 7, wherein decompressing the selected grid cells includes performing ray tracing comprising:
marching rays until intersections with surfaces in the virtual scene are found; and
and for each of the rays for which a respective intersection is found:
determining a pixel color based, at least in part, on a product of a distance of a ray point at the respective intersection to a closest depth sample from one of the cameras and a distance of the ray point to the one of the cameras.

### Claim 11

A non-transitory computer-readable storage medium storing a program, which, when executed by a processor performs operations comprising:
determining placements for a plurality of virtual cameras in a virtual scene, wherein the placements for the plurality of virtual cameras provide full motion light field visibility;
rendering the virtual scene using the virtual cameras at the determined placements, wherein the rendering produces color data and depth data; and
compressing the color data and the depth data.

### Claim 12 (depends on 11)

The computer-readable storage medium of claim 11, wherein the placements for the plurality of virtual cameras are determined based, at least in part, on an objective function which includes a term that penalizes camera redundancy.

### Claim 13 (depends on 12)

The computer-readable storage medium of claim 12, wherein:
the virtual scene includes a specular lighting environment; and
the term in the objective function that penalizes camera redundancy includes an angular weight term.

### Claim 14 (depends on 11)

The computer-readable storage medium of claim 11, wherein compressing the color data includes temporally compressing the color data, the temporal compression of the color data comprising:
dividing each of a plurality of rendered video frames into a plurality of grid cells; and
for each grid cell of the plurality of grid cells:
determining, using an optimization process, a smallest selection of keyframes of the grid cell that can be used to derive other frames of the grid cell, and
storing the determined keyframes and an array including parameter values for each of the other frames of the grid cell, wherein each of the parameter values indicates, for a corresponding one of the other frames, interpolations of closest keyframes to the one of the other frames forward and backward in time.

### Claim 15 (depends on 11)

The computer-readable storage medium of claim 11, wherein compressing the depth data includes temporally compressing the depth data, the temporal compression of the depth data comprising:
determining one or more keyframes from rendered depth frames; and
storing depth frames from the rendered depth frames that are not the one or more keyframes as P-frames that each encode differences to a last keyframe.

### Claim 16 (depends on 15)

The computer-readable storage medium of claim 15, wherein the differences to the last keyframe are encoded using a set of axis-aligned bounding boxes (AABBs) in a leaf level of an AABB hierarchy.

### Claim 17 (depends on 11)

The computer-readable storage medium of claim 11, the operations further comprising:
decompressing at least a portion of the compressed data; and
reconstructing one or more video frames using the decompressed portion of the compressed data.

### Claim 18 (depends on 17)

The computer-readable storage medium of claim 17, wherein decompressing at least the portion of the compressed data includes:
selecting a plurality of the virtual cameras using one or more heuristics;
selecting grid cells from the selected cameras that are visible to a viewer; and
decompressing the selected grid cells.

### Claim 19 (depends on 18)

The computer-readable storage medium of claim 18, wherein the one or more heuristics include at least one of:
a distance-based heuristic which gives higher priority to viewpoints with closer distances to an eye location;
an angle-based heuristic which gives higher priority to viewpoints with smaller angles to the eye location;
an angle-based culling heuristic which culls any viewpoint forming an angle with another higher priority viewpoint that is smaller than a given threshold; and
a performance-based culling heuristic which culls viewpoints whose runtime performance is below a predefined threshold.

### Claim 20

A system, comprising:
a processor; and
a memory, wherein the memory includes an application program configured to perform operations comprising:
determining placements for a plurality of virtual cameras in a virtual scene, wherein the placements for the plurality of virtual cameras provide full motion light field visibility,
rendering the virtual scene using the virtual cameras at the determined placements, wherein the rendering produces color data and depth data, and
compressing the color data and the depth data.