# US 12,112,487 B2 · Camera tracking via dynamic perspectives > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,487 B2 - Document ID: US12112487B2 - Status: Granted - Kind: B2 - Application number: 17/968,432 - Filed: 2022-10-18 - Published: 2023-04-20 - Granted: 2024-10-08 - Earliest claimed: 2021-10-20 - Assignees: NantStudios, LLC - Applicants: NantStudios, LLC - Inventors: Matheen Siddiqui - Web page: https://patentgrep.com/patent/US12112487B2 ## CPC - G06T 7/248: {involving reference images or patches} - G06T 7/292: Multi-camera tracking - G06T 7/74: {involving reference images or patches} - G06T 13/20: Three-dimensional [3D] animation - G06T 15/20: Perspective computation - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - H04N 5/2224: {related to virtual studio applications} - G06T 2207/30244: Camera pose - H04N 5/275: Generation of keying signals - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A computer system may identify a first position of a physical camera corresponding to a first time period. The computer system may render a first virtual scene for the first time period. The system may project the first scene onto a display surface to determine a first rendered image for the first time period. The computer system may receive a first camera image of the display surface from the camera during the first time period. The system may determine a first corrected position of the camera by comparing the first rendered image to the first camera image. The system may predict a second position of the camera corresponding to a second time period. The computer system may render a second virtual scene for the second time period. The system may project the second virtual scene onto the display surface to determine a second rendered image for the second time period. ## Claims ### Claim 1 A method for implementing camera tracking in a virtual production environment, the method comprising performing, by a computer system communicably coupled with a physical camera that is movable: identifying a first position of the physical camera corresponding to a first time period; rendering, using an animation engine, a first virtual scene for the first time period; projecting the first virtual scene onto a display surface to determine a first rendered image for the first time period, the projecting of the first virtual scene consistent with the first position of the physical camera; storing the first rendered image in a frame buffer for displaying on the display surface; receiving a first camera image of the display surface, the first camera image obtained using the physical camera during the first time period; determining a first corrected position of the physical camera by comparing the first rendered image to the first camera image; predicting, using the first corrected position, a second position of the physical camera corresponding to a second time period in which a second rendered image is to be displayed on a display surface; rendering, using the animation engine, a second virtual scene for the second time period; and projecting the second virtual scene onto the display surface to determine a second rendered image for the second time period, the projecting of the second virtual scene consistent with the second position of the physical camera. ### Claim 2 (depends on 1) The method of claim 1, wherein comparing the first rendered image to the first camera image includes: identifying a first pixel location of an object in the first rendered image and a second pixel location of the object in the first camera image; determining a difference between the first pixel location and the second pixel location; and applying the difference to the first position to obtain the first corrected position. ### Claim 3 (depends on 1) The method of claim 1, wherein predicting the second position uses information from a predetermined choreographic file for a final video that is to be obtained. ### Claim 4 (depends on 1) The method of claim 1, wherein the first position of the physical camera is identified using an initial image from the physical camera of the display screen. ### Claim 5 (depends on 4) The method of claim 4, identifying the first position of the physical camera includes: storing a model that maps images to physical positions; and inputting the initial image to the model. ### Claim 6 (depends on 1) The method of claim 1, wherein predicting the second position of the physical camera uses information from a history of positions of a physical camera. ### Claim 7 (depends on 1) The method of claim 1, wherein predicting the second position of the physical camera includes: storing a trained model that predicts a future position from a current position; and inputting the first position to the model. ### Claim 8 (depends on 7) The method of claim 7, wherein inputting the first position to the model includes inputting a set of movement data for the physical camera at the first position. ### Claim 9 (depends on 8) The method of claim 8, wherein the set of movement data includes at least one of velocity, acceleration, jerk, snap, crackle, and pop. ### Claim 10 A computer product comprising a non-transitory computer readable medium storing a plurality of instructions that when executed control a computer system to perform: identifying a first position of a physical camera corresponding to a first time period; rendering, using an animation engine, a first virtual scene for the first time period; projecting the first virtual scene onto a display surface to determine a first rendered image for the first time period, the projecting of the first virtual scene consistent with the first position of the physical camera; storing the first rendered image in a frame buffer for displaying on the display surface; receiving a first camera image of the display surface, the first camera image obtained using the physical camera during the first time period; determining a first corrected position of the physical camera by comparing the first rendered image to the first camera image; predicting, using the first corrected position, a second position of the physical camera corresponding to a second time period in which a second rendered image is to be displayed on a display surface; rendering, using the animation engine, a second virtual scene for the second time period; and projecting the second virtual scene onto the display surface to determine a second rendered image for the second time period, the projecting of the second virtual scene consistent with the second position of the physical camera. ### Claim 11 (depends on 10) The computer product of claim 10, wherein comparing the first rendered image to the first camera image includes: identifying a first pixel location of an object in the first rendered image and a second pixel location of the object in the first camera image; determining a difference between the first pixel location and the second pixel location; and applying the difference to the first position to obtain the first corrected position. ### Claim 12 (depends on 10) The computer product of claim 10, wherein predicting the second position uses information from a predetermined choreographic file for a final video that is to be obtained. ### Claim 13 (depends on 10) The computer product of claim 10, wherein the first position of the physical camera is identified using an initial image from the physical camera of the display screen. ### Claim 14 (depends on 13) The computer product of claim 13, identifying the first position of the physical camera includes: storing a model that maps images to physical positions; and inputting the initial image to the model. ### Claim 15 (depends on 10) The computer product of claim 10, wherein predicting the second position of the physical camera uses information from a history of positions of a physical camera. ### Claim 16 A system comprising: a physical camera; a non-transitory computer readable medium; and one or more processors communicably coupled with the non-transitory computer readable medium and configured to perform: identifying a first position of the physical camera corresponding to a first time period; rendering, using an animation engine, a first virtual scene for the first time period; projecting the first virtual scene onto a display surface to determine a first rendered image for the first time period, the projecting of the first virtual scene consistent with the first position of the physical camera; storing the first rendered image in a frame buffer for displaying on the display surface; receiving a first camera image of the display surface, the first camera image obtained using the physical camera during the first time period; determining a first corrected position of the physical camera by comparing the first rendered image to the first camera image; predicting, using the first corrected position, a second position of the physical camera corresponding to a second time period in which a second rendered image is to be displayed on a display surface; rendering, using the animation engine, a second virtual scene for the second time period; and projecting the second virtual scene onto the display surface to determine a second rendered image for the second time period, the projecting of the second virtual scene consistent with the second position of the physical camera. ### Claim 17 (depends on 16) The system of claim 16, wherein predicting the second position of the physical camera uses information from a history of positions of a physical camera. ### Claim 18 (depends on 16) The system of claim 16, wherein predicting the second position of the physical camera includes: storing a trained model that predicts a future position from a current position; and inputting the first position to the model. ### Claim 19 (depends on 18) The system of claim 18, wherein inputting the first position to the model includes inputting a set of movement data for the physical camera at the first position. ### Claim 20 (depends on 19) The system of claim 19, wherein the set of movement data includes at least one of velocity, acceleration, jerk, snap, crackle, and pop.