Back to results
US 11,606,546 B1Granted
Perspective based green screening
AssigneeTanzle, Inc.
InventorsNancy L. Clemens, Michael A. Vesely
Life of this patent
Estimating the term…
Abstract
A method of generating a composite image includes capturing a video image of a physical scene with a camera, identifying a green-screen region within the video image, identifying a viewpoint and a position and/or orientation of the green-screen region relative to the viewpoint, and generating a modified video image rendered from the viewpoint onto a display surface in which the green-screen region is replaced with an image of a virtual object. The image of the virtual object is generated by projection rendering of a model of the virtual object based on the position and/or orientation of the green-screen region relative to the viewpoint such that the virtual object is constrained within the green-screen region.
Claims21
A method comprising:
- receiving a stereoscopic video image of a physical scene captured by a camera, the physical scene comprising an object having a tracking component;
- storing data providing a model defining (i) dimensions of a three-dimensional green-screen region and (ii) a location and/or orientation of the three-dimensional green-screen region relative to the object;
- receiving, from the tracking component, data identifying a location of the object;
- identifying a three-dimensional region of space in the physical scene for the three-dimensional green-screen region based on (i) the location received from the tracking component and (ii) the data defining dimensions of the three-dimensional green-screen region and the location and/or orientation of the three-dimensional green-screen region relative to the object;
- identifying an eyepoint pair in the physical scene;
- identifying, for each eyepoint of the eyepoint pair, a position and/or orientation of the three-dimensional green-screen region relative to the eyepoint;
- and
- rendering a modified stereoscopic video image onto a display surface in which the three-dimensional green-screen region is replaced with a depiction of a virtual three-dimensional object, the rendering comprising:
- obtaining a three-dimensional model of the virtual three-dimensional object,
- determining a three-dimensional boundary of the three-dimensional green-screen region,
- determining, for each eyepoint of the eyepoint pair, a respective different perspective of the virtual three-dimensional object based on (i) the three-dimensional model of the virtual three-dimensional object and (ii) the position and/or orientation of the three-dimensional green-screen region relative to the eyepoint, and
- for each eyepoint of the eyepoint pair, performing a projection rendering of the three-dimensional model of the virtual three-dimensional object based on the determined perspective of the eyepoint such that the depiction of the virtual three-dimensional object is constrained to be within the determined three-dimensional boundary of the three-dimensional green-screen region.
9 more elements
A computer program product comprising a non-transitory computer readable medium having instructions for causing one or more processors to:
- receive a stereoscopic video image of a physical scene captured by a camera, the physical scene comprising an object having a tracking component;
- store data providing a model defining (i) dimensions of a three-dimensional green-screen region and (ii) a location and/or orientation of the three-dimensional green-screen region relative to the object;
- receive, from the tracking component, data identifying a location of the object;
- identify a three-dimensional region of space in the physical scene for the three-dimensional green-screen region based on (i) the location received from the tracking component and (ii) the data defining dimensions of the three-dimensional green-screen region and the location and/or orientation of the three-dimensional green-screen region relative to the object;
- identify an eyepoint pair in the physical scene;
- identify, for each eyepoint of the eyepoint pair, a position and/or orientation of the three-dimensional green-screen region relative to the eyepoint;
- and
- render a modified stereoscopic video image onto a display surface in which the three-dimensional green-screen region is replaced with a depiction of a virtual three-dimensional object, the instructions to render comprising instructions to:
- obtain a three-dimensional model of the virtual three-dimensional object,
- determine a three-dimensional boundary of the three-dimensional green-screen region,
- determine, for each eyepoint of the eyepoint pair, a respective different perspective of the virtual three-dimensional object based on (i) the three-dimensional model of the virtual three-dimensional object and (ii) the position and/or orientation of the three-dimensional green-screen region relative to the eyepoint, and
- for each eyepoint of the eyepoint pair, perform a projection rendering of the three-dimensional model of the virtual three-dimensional object based on the determined perspective of the eyepoint such that the depiction of the virtual three-dimensional object is constrained to be within the determined three-dimensional boundary of the three-dimensional green-screen region.
9 more elements
A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
- receiving a stereoscopic video image of a physical scene captured by a camera, the physical scene comprising an object having a tracking component;
- storing data providing a model defining (i) dimensions of a three-dimensional green-screen region and (ii) a location and/or orientation of the three-dimensional green-screen region relative to the object;
- receiving, from the tracking component, data identifying a location of the object;
- identifying a three-dimensional region of space in the physical scene for the three-dimensional green-screen region based on (i) the location received from the tracking component and (ii) the data defining dimensions of the three-dimensional green-screen region and the location and/or orientation of the three-dimensional green-screen region relative to the object;
- identifying an eyepoint pair in the physical scene;
- identifying, for each eyepoint of the eyepoint pair, a position and/or orientation of the three-dimensional green-screen region relative to the eyepoint;
- and
- rendering a modified stereoscopic video image onto a display surface in which the three-dimensional green-screen region is replaced with a depiction of a virtual three-dimensional object, the rendering comprising:
- obtaining a three-dimensional model of the virtual three-dimensional object,
- determining a three-dimensional boundary of the three-dimensional green-screen region,
- determining, for each eyepoint of the eyepoint pair, a respective different perspective of the virtual three-dimensional object based on (i) the three-dimensional model of the virtual three-dimensional object and (ii) the position and/or orientation of the three-dimensional green-screen region relative to the eyepoint, and
- for each eyepoint of the eyepoint pair, performing a projection rendering of the three-dimensional model of the virtual three-dimensional object based on the determined perspective of the eyepoint such that the depiction of the virtual three-dimensional object is constrained to be within the determined three-dimensional boundary of the three-dimensional green-screen region.
9 more elements
Dependencies are inferred from the claim text.