# US 12,217,363 B2 · Systems and methods for capturing, transporting, and reproducing three-dimensional simulations as interactive volumetric displays

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,363 B2
- Document ID: US12217363B2
- Status: Granted
- Kind: B2
- Application number: 18/371,815
- Filed: 2023-09-22
- Published: 2024-06-06
- Granted: 2025-02-04
- Earliest claimed: 2022-09-14
- Assignees: Liv, Inc.
- Applicants: Liv, Inc.
- Inventors: Jaroslav Stehlik; Alexander Brook Perry; Arthur Louis Brainville; Steffan Robert William Donal; Bunta Adrian-Nicolae; Omar Mohamed Ali Mudhir; Ajinyad Karwan Shewki
- Web page: https://patentgrep.com/patent/US12217363B2

## CPC

- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06T 7/50: Depth or shape recovery
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T 7/90: Determination of colour characteristics
- G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00)
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04N 13/351: for displaying simultaneously
- H04N 19/597: specially adapted for multi-view video sequence encoding
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A system, method, and device are provided for creating a data processing pipeline to capture, transport, and interactively display in both recorded and real time any 3D volumetric simulation that was generated by a separate application. The system, method, and device enabling the interactive viewing of virtual and mixed reality simulations by a plurality of users.

## Claims

### Claim 1

A method for processing mixed reality 3D volumetric environment data comprised of a simulated portion and a real-world portion so that the mixed reality 3D volumetric environment can be viewed interactively by at least one final user, the method using a plurality of virtual cameras in the simulated portion and at least one real-world video camera focused in the same position and orientation in the mixed reality 3D environment as at least one virtual camera, the method comprising:
(a) capturing from the virtual cameras the simulated portion of the mixed reality 3D volumetric environment with at least the following data in a format using time tags:
(i) intrinsic data from each of the virtual cameras on a frame-by-frame basis,
(ii) extrinsic data from each of the virtual cameras on a frame-by-frame basis, and
(iii) depth data from each of the virtual cameras on a frame-by-frame basis;
(b) capturing from the at least one real-world video camera portion of the mixed reality 3D volumetric environment at least the following data in a format using time tags:
(i) intrinsic data from the at least one real-world video camera on a frame-by-frame basis of a real-world actor in the field of view of the real-world video camera, and
(ii) extrinsic data from the at least one real-world video camera on a frame-by-frame basis, and
(c) compressing the time-tagged data captured in steps (a) and (b) by:
(i) compressing the frame-by-frame intrinsic and extrinsic data from each of the virtual capture cameras and each of the real-world cameras, and
(ii) compressing the frame-by-frame depth data from each of the virtual capture cameras and each of the real-world cameras; (d) preparing the time-tagged data compressed in step (c) for viewing by the at least one final user by:
(i) tessellate shading each virtual camera frame as a grid of uniformly spaced unconnected quadratures,
(ii) converting each of the quadratures into a specific location within the simulated 3D volumetric environment,
(iii) positioning each vertex of the quadratures in the 3D volumetric environment based on an X/Y position in each virtual camera's frame and depth, the depth being determined using the depth data for each frame, and
(iv) isolating the real-world actor images in the field of view of the real-world video camera from any real-world background and overlaying the isolated real-world actor images over the quadratures in the same position and orientation in the simulated portion of the mixed reality 3D environment as the at least one real-world video camera, thereby enabling the mixed reality 3D volumetric environment to be viewed interactively by the at least one final user.

### Claim 2 (depends on 1)

The method of claim 1 wherein step (a) further comprises capturing:
(iv) depth data from the at least one real-world video camera on a frame-by-frame basis.

### Claim 3 (depends on 1)

The method of claim 1 wherein step (b) further comprises capturing:
(iii) ambient audio from the at least one real-world actor.

### Claim 4 (depends on 1)

The method of claim 1 wherein the real-world background in the field of view of the real-world video camera is a homogeneous surface.

### Claim 5 (depends on 4)

The method of claim 4 wherein the real-world background in the field of view of the real-world video camera is a homogeneous green screen.

### Claim 6 (depends on 1)

The method of claim 1 wherein step (b) further comprises capturing:
(iii) ambient audio from the simulated portion of the mixed reality 3D volumetric environment.

### Claim 7 (depends on 1)

The method of claim 1 wherein the frame-by-frame depth data compression is executed by reducing the bit depth of the depth data.

### Claim 8 (depends on 1)

The method of claim 1 wherein the frame-by-frame depth data compression is executed via a bias skewed towards portions of objects of interest.

### Claim 9 (depends on 8)

The method of claim 8 wherein the bias skewed towards portions of objects of interest data compression is a ramp non-linear mathematical function.

### Claim 10 (depends on 1)

The method of claim 1 wherein corners of the quadratures are connected to corners of neighboring quadratures when the difference between the depth of neighboring quadrature corners is less than a threshold value.

### Claim 11 (depends on 1)

The method of claim 1 wherein the captured time-tagged data is recorded into non-volatile memory for later playback by the at least one final user.

### Claim 12 (depends on 1)

The method of claim 1 wherein an equirectangular spherical view of the simulated environment is also captured in addition to the captured virtual camera frames.

### Claim 13

A method for interactively viewing a stream of 3D volumetric environment data by at least one final user using time tagged frame-by-frame intrinsic from a plurality of cameras, the viewing stream data comprised of intrinsic data comprised of at least a matrix of color pixels from each of the plurality of cameras, extrinsic data comprised of at least an X/Y/Z position within the 3D volumetric environment for each of the plurality of cameras, and associated depth pixel data for each color pixel within a matrix, the method comprising:
(a) the final user selecting a viewing field-of-view projection matrix that resides within the 3D volumetric environment;
(b) mapping each depth pixel and associated intrinsic color pixel to a quadrature with an X/Y/Z position in the 3D volumetric environmental space that appears within the final user's selected field-of-view projection matrix;
(c) resolving corners of each quadrature by comparing each quadrature's corner depth to adjacent quadratures and connecting the adjacent quadratures, thereby creating a single composite quadrature, when the difference between the depth of adjacent quadrature corners is less than an a priori threshold value; and
(d) applying an appropriate color to each composite quadrature based on the colors of the pixels comprising the composite quadrature;
thereby enabling the streamed 3D volumetric environment data to be viewed interactively by the at least one final user.

### Claim 14 (depends on 13)

The method of claim 13 wherein bilinear filtering is applied to determine the appropriate color.

### Claim 15 (depends on 14)

The method of claim 14 wherein the bilinear filtering determining the appropriate color is a function of depth steepness, with a smaller valued depth steepness receiving more filtering.

### Claim 16 (depends on 13)

The method of claim 13 wherein the final user employs a virtual laser pointer tool to select the viewing camera's field-of-view projection matrix.

### Claim 17 (depends on 16)

The method of claim 16 wherein the virtual laser pointer tool includes an adjustable zoom feature.

### Claim 18 (depends on 13)

The method of claim 13 wherein the viewed 3D volumetric environment data includes audio.

### Claim 19 (depends on 18)

The method of claim 18 wherein the audio is played back in two channels.

### Claim 20 (depends on 13)

The method of claim 13 wherein the stream of 3D volumetric environment data is decompressed prior to processing.

### Claim 21 (depends on 13)

The method of claim 13 wherein the final user selected viewing field-of-view projection matrix is an exact replica of one of the virtual camera perspectives.
