# US 12,112,427 B2 · High-definition real-time view synthesis > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,427 B2 - Document ID: US12112427B2 - Status: Granted - Kind: B2 - Application number: 17/898,136 - Filed: 2022-08-29 - Published: 2023-03-02 - Granted: 2024-10-08 - Earliest claimed: 2021-08-27 - Assignees: Snap Inc. - Applicants: Snap Inc. - Inventors: Numair Khalil Ullah Khan; Gurunandan Krishnan Gorumkonda; Shree K. Nayar; Yicheng Wu - Web page: https://patentgrep.com/patent/US12112427B2 ## CPC - G06T 15/205: {Image-based rendering} - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 7/55: from multiple images - G06T 15/04: Texture mapping - G06T 15/20: Perspective computation - G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation} - H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking - H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Images of a scene are received. The images represent viewpoints corresponding to the scene. A pixel map of the scene is computed based on the plurality of images. Multi-plane image (MPI) layers from the pixel map are extracted in real-time. The MPI layers are aggregated. The scene is rendered from a novel viewpoint based on the aggregated MPI layers. ## Claims ### Claim 1 A method comprising: receiving a plurality of images captured from a reference viewpoint of a scene, each image of the plurality of images representing a view of the scene at a predefined depth from the reference viewpoint; generating a two-dimensional (2D) pixel map of the scene based on the plurality of images; determining one or more multi-plane image (MPI) layers from the 2D pixel map in real-time; dividing the one or more MPI layers into a plurality of MPI regions, wherein each MPI region of the plurality of MPI regions has a different set of depth planes than all other MPI regions of the plurality of MPI regions, and wherein a number of depth planes of the one or more MPI layers is larger than a number of depth planes of each MPI region of the plurality of MPI regions; constructing a textured three-dimensional mesh based on the plurality of MPI regions; rendering the scene from a novel viewpoint based on the textured three-dimensional mesh. ### Claim 2 (depends on 1) The method of claim 1, wherein the scene is rendered at a resolution of 1080p. ### Claim 3 (depends on 1) The method of claim 1, wherein receiving the plurality of images comprises capturing the plurality of images using at least one image capture device. ### Claim 4 (depends on 3) The method of claim 3, wherein the at least one image capture device is a mobile device. ### Claim 5 (depends on 1) The method of claim 1, wherein at least one viewpoint of the plurality of viewpoints is related to at least one other viewpoint of the plurality of viewpoints. ### Claim 6 (depends on 1) The method of claim 1, further comprising: receiving a video of the scene, wherein receiving the plurality of images comprises extracting the plurality of images of the scene based on the received video. ### Claim 7 (depends on 1) The method of claim 1, wherein the plurality of images are RGBA images. ### Claim 8 A computing apparatus, the computing apparatus comprising: a processor; and a memory storing instructions that, when executed by the processor, configure the apparatus to: receive a plurality of images captured from a reference viewpoint of a scene, each image of the plurality of images representing a view of the scene at a predefined depth from the reference viewpoint; generate a two-dimensional (2D) pixel map of the scene based on the plurality of images; determine one or more of multi-plane image (MPI) layers from the 2D pixel map in real-time; divide the one or more MPI layers into a plurality of MPI regions, wherein each MPI region of the plurality of MPI regions has a different set of depth planes than all other MPI regions of the plurality of MPI regions, and wherein a number of depth planes of the one or more MPI layers is larger than a number of depth planes of each MPI region of the plurality of MPI regions; construct a textured three-dimensional mesh based on the plurality of MPI regions; render the scene from a novel viewpoint based on the textured three-dimensional mesh. ### Claim 9 (depends on 8) The computing apparatus of claim 8, wherein the apparatus is further configured to render the scene at a resolution of 1080p. ### Claim 10 (depends on 8) The computing apparatus of claim 8, wherein the computing apparatus further comprises at least one image capture device. ### Claim 11 (depends on 8) The computing apparatus of claim 8, wherein the computing apparatus is a mobile device. ### Claim 12 (depends on 8) The computing apparatus of claim 8, wherein at least one viewpoint of the plurality of viewpoints is related to at least one other viewpoint of the plurality of viewpoints. ### Claim 13 (depends on 8) The computing apparatus of claim 8, wherein the apparatus is further configured to: receive a video of the scene; and wherein receiving a plurality of images of the scene comprises extracting the plurality of images of the scene based on the received video. ### Claim 14 A non-transitory computer-readable storage medium including instructions that, when processed by a computer, configure the computer to perform: receiving a plurality of images captured from a reference viewpoint of a scene, each image of the plurality of images representing a view of the scene at a predefined depth from the reference viewpoint; generating a two-dimensional (2D) pixel map of the scene based on the plurality of images; determining one or more of multi-plane image (MPI) layers from the 2D pixel map in real-time; dividing the one or more MPI layers into a plurality of MPI regions, wherein each MPI region of the plurality of MPI regions has a different set of depth planes than all other MPI regions of the plurality of MPI regions, and wherein a number of depth planes of the one or more MPI layers is larger than a number of depth planes of each MPI region of the plurality of MPI regions; constructing a textured three-dimensional mesh based on the plurality of MPI regions; rendering the scene from a novel viewpoint based on the textured three-dimensional mesh.