Quantized efficient encoding for streaming free-viewpoint videos
AssigneeNVIDIA Corporation
InventorsAmrita Mazumdar, Tianye Li, Shalini De Mello, Sharath Girish
Life of this patent
Estimating the term…
Abstract
Systems and methods are provided for streaming free-viewpoint videos (FVV) of a dynamic 3D scene. Each time step of the dynamic 3D scene is represented as a set of 3D Gaussians, each 3D Gaussian having a set of Gaussian attributes. Gaussian residuals are encoded for every time step. In at least one embodiment, position residuals are sparsified and non-position attribute residuals are quantized, thereby achieving compression factors without sacrificing reconstruction quality.
Claims20
A method for encoding a free-viewpoint video (FVV), the method comprising:
- receiving, as input, a first multi-view frame of a scene corresponding to a first time;
- generating, based on the first multi-view frame, a plurality of three-dimensional (3D) Gaussians that collectively represent the scene at the first time, wherein each 3D Gaussian includes a position attribute and one or more non-position attributes;
- receiving, as further input, a second multi-view frame of the scene corresponding to a second time;
- and
- determining, for each 3D Gaussian of the plurality of 3D Gaussians, a position residual and one or more non-position residuals, the determining comprising:
- modeling the position residuals using a sparsity framework,
- modeling the non-position residuals using a quantization framework, and
- learning the position residuals and the non-position residuals using machine learning techniques.
A system for encoding a free-viewpoint video (FVV), the system comprising:
- processing circuitry configured to:
- receive, as input, a first multi-view frame of a scene corresponding to a first time;
- generate, based on the first multi-view frame, a plurality of three-dimensional (3D) Gaussians that collectively represent the scene at the first time, wherein each 3D Gaussian includes a position attribute and one or more non-position attributes,
- receive, as further input, a second multi-view frame of the scene corresponding to a second time, and
- determine, for each 3D Gaussian of the plurality of 3D Gaussians, a position residual and one or more non-position residuals, the processing circuitry being configured to determine the position residuals and the non-position residuals by:
- modeling the position residuals using a sparsity framework,
- modeling the non-position residuals using a quantization framework, and
- learning the position residuals and the non-position residuals using machine learning techniques;
- and
- one or more memories configured to store the first multi-view frame, the plurality of 3D Gaussians, the second multi-view frame, and the position residuals and the non-position residuals.
A non-transitory computer-readable media storing processor-executable instructions for encoding a free-viewpoint video (FVV) that, when executed by processing circuitry, cause the processing circuitry to perform a method comprising:
- receiving, as input, a first multi-view frame of a scene corresponding to a first time;
- generating, based on the first multi-view frame, a plurality of three-dimensional (3D) Gaussians that collectively represent the scene at the first time, wherein each 3D Gaussian includes a position attribute and one or more non-position attributes;
- receiving, as further input, a second multi-view frame of the scene corresponding to a second time;
- and
- determining, for each 3D Gaussian of the plurality of 3D Gaussians, a position residual and one or more non-position residuals, the determining comprising:
- modeling the position residuals using a sparsity framework,
- modeling the non-position residuals using a quantization framework, and
- learning the position residuals and the non-position residuals using machine learning techniques.
Dependencies are inferred from the claim text.