# US 12,112,426 B2 · Augmenting a view of a real-world environment with a view of a volumetric video object > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,426 B2 - Document ID: US12112426B2 - Status: Granted - Kind: B2 - Application number: 17/784,883 - Filed: 2020-12-15 - Published: 2023-01-12 - Granted: 2024-10-08 - Earliest claimed: 2019-12-16 - Assignees: Fraunhofer-Gessellschaft zur Forderung der angewandten Forschung e.V. - Applicants: Fraunhofer-Gesellschaft Zur FöRderung Der Angewandten Forschung E.V. - Inventors: Cornelius Hellge; Thomas Schierl; Peter Eisert; Anna Hilsmann; Robert Skupin; Yago Sanchez; Jangwoo Son; Gurdeep Singh Bhullar; Serhan Gül; Dimitri Podborski - Web page: https://patentgrep.com/patent/US12112426B2 ## CPC - G06T 15/205: {Image-based rendering} - G06T 7/194: involving foreground-background segmentation - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 15/04: Texture mapping - G06T 15/08: Volume rendering - G06T 15/60: Shadow generation - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T 2207/20112: Image segmentation details - G06T 2210/08: Bandwidth reduction - G06T 2215/16: Using real world measurements to influence rendering - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method of augmenting a view of a real-world environment with a view of a volumetric video object on a user device is disclosed. The method includes determining a current pose information (CPI) indicating a current pose of the view of the real-world environment and a desired pose of the volumetric video object in the real-world environment. The method further includes sending the CPI to a remote server. The method further includes receiving a rendered view of the volumetric video object that has been rendered in accordance with the CPI from the remote server. The method also includes augmenting the view of the real-world environment by at least mapping the rendered view of the volumetric video object onto a planar mapping surface arranged according to the desired position of the volumetric video object. ## Claims ### Claim 1 A method of augmenting a view of a real-world environment with a view of a volumetric video object on a user device, the method comprising the steps of: determining a current pose information (CPI) indicating a current pose of the view of the real-world environment and a desired pose of the volumetric video object in the real-world environment; sending the current pose information to a remote server; receiving a rendered view of the volumetric video object that has been rendered in accordance with the current pose information from the remote server; and augmenting the view of the real-world environment by at least mapping the rendered view of the volumetric video object onto a planar mapping surface arranged according to the desired position of the volumetric video object; wherein the planar mapping surface is oriented such that the rendered view of the volumetric video object is projected into the view of the real-world environment substantially free of perspective distortion. ### Claim 2 (depends on 1) The method as defined in claim 1, wherein the rendered view of the volumetric video object comprises at least one background region that does not show the volumetric video object, and wherein the augmenting step comprises setting the at least one background region to be transparent. ### Claim 3 (depends on 2) The method as defined in claim 2, wherein the rendered view of the volumetric video object comprises a transparency map that indicates the at least one background region is indicated in the rendered view of the volumetric video object by a predefined color, and wherein the augmenting step further comprises determining the at least one background region based on the transparency map or the predefined color. ### Claim 4 (depends on 2) The method as defined in claim 2, wherein the augmenting step further comprises segmenting the volumetric video object as shown in the rendered view of the volumetric video object and determining the at least one background region based on the segmentation of the volumetric video object. ### Claim 5 (depends on 1) The method as defined in claim 1, wherein the size and/or shape of the planar mapping surface is dynamically set based on the size and/or shape of the volumetric video object as shown in the rendered view of the volumetric video object. ### Claim 6 (depends on 1) The method as defined in claim 1, wherein the receiving step further comprises receiving an approximate volume information indicating the approximate volume of the volumetric video object, and wherein the augmenting step further comprises using the approximate volume information for one or more of collision avoidance, correcting a lighting of the rendered view of the volumetric video object, generating at least one shadow of the volumetric video object in the real-world environment, and generating at least one reflection of the volumetric video object in the real-world environment. ### Claim 7 (depends on 6) The method as defined in claim 6, wherein the approximate volume information comprises first approximate volume information for visible parts of the volumetric video object as shown in the rendered view of the volumetric video object and/or second approximate volume information for non-visible parts of the volumetric video object as shown in the rendered view of the volumetric video object. ### Claim 8 (depends on 7) The method as defined in claim 7, wherein the first approximate volume information and/or the second approximate volume information comprises at least one of a depth map, mesh data, and one or more geometric primitives that approximate the volume of the volumetric video object. ### Claim 9 (depends on 6) The method as defined in claim 6, wherein the determining step further comprises determining a reflection information indicating at least one reflection of the volumetric video object in the real-world environment, wherein the sending step further comprises sending the reflection information to the remote server, wherein the receiving step further comprises receiving at least one rendered view of the at least one reflection that has been rendered taking into account the reflection information from the remote server, and wherein the augmenting step further comprises generating the at least one reflection of the volumetric video object in the real-world environment based on the at least one rendered view of the at least one reflection. ### Claim 10 (depends on 1) The method as defined in claim 1, wherein the determining step further comprises determining a lighting information indicating a lighting of the real-world environment, wherein the sending step further comprises sending the lighting information to the remote server, and wherein the rendered view of the volumetric video object has been rendered taking into account the lighting information. ### Claim 11 (depends on 1) The method as defined in claim 1, wherein the sending step further comprises sending the view of the real-world environment to the remote server, wherein the rendered view of the volumetric video object has been rendered taking into account a lighting of the real-world environment, and/or wherein the receiving step further comprises receiving at least one rendered view of at least one shadow of the volumetric video object in the real-world environment, and/or wherein the receiving step further comprises receiving at least one rendered view of at least one reflection that has been rendered taking into account at least one reflection of the volumetric video object in the real-world environment, and/or wherein the augmenting step comprises generating the at least one shadow of the volumetric video object in the real-world environment based on the at least one rendered view of the at least one shadow, and/or wherein the augmenting step comprises generating the at least one reflection of the volumetric video object in the real-world environment based on the at least one rendered view of the at least one reflection, wherein the lighting and/or the at least one shadow and/or the at least one reflection has been determined based on the view of the real-world environment. ### Claim 12 (depends on 11) The method as defined in claim 11, wherein the rendered view of the volumetric video object is received in a packed texture in which it is packed together with a rendered view of a further volumetric video object and/or the at least one rendered view of the at least one shadow and/or the at least one rendered view of the at least one reflection. ### Claim 13 (depends on 12) The method as defined in claim 12, wherein the receiving further comprises receiving metadata comprising unpack information indicating how to unpack the packed texture into the different rendered views. ### Claim 14 (depends on 1) The method as defined in claim 1, wherein the view of the real-world environment and the rendered view of the volumetric video object are stereoscopic views comprising two different views for the two eyes of a user, and wherein the augmenting step comprises at least mapping each of the two different views of the rendered view of the volumetric video object onto a planar mapping surface. ### Claim 15 (depends on 1) The method as defined in claim 1, wherein the view of the real-world environment is a stereoscopic view comprising two different views for the two eyes of a user, wherein the receiving step comprises receiving stereoscopic view generation information for rendering a stereoscopic view of the volumetric video object from the rendered view of the volumetric video object, and wherein the augmenting step further comprises generating the stereoscopic view of the volumetric video object from the rendered view of the volumetric video object in accordance with the stereoscopic view generation information and mapping each of the two different views of the generated stereoscopic view of the volumetric video object onto a planar mapping surface. ### Claim 16 A user device for augmenting a view of a real-world environment with a view of a volumetric video object, comprising: a determining unit for determining a current pose information (CPI) indicating a current pose of the view of the real-world environment and a desired pose of the volumetric video object in the real-world environment; a sending unit for sending the CPI to a remote server; a receiving unit for receiving a rendered view of the volumetric video object that has been rendered in accordance with the CPI from the remote server; and an augmenting unit for augmenting the real-world environment by at least mapping the rendered view of the volumetric video object onto a planar mapping surface arranged according to the desired position of the volumetric video object; wherein the planar mapping surface is oriented such the at the rendered view of the volumetric video object is projected into the view of the real-world environment substantially free of perspective distortion.