# US 12,217,375 B2 · System for mixing or compositing in real-time, computer generated 3D objects and a video feed from a film camera > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,375 B2 - Document ID: US12217375B2 - Status: Granted - Kind: B2 - Application number: 18/230,721 - Filed: 2023-08-07 - Published: 2023-11-30 - Granted: 2025-02-04 - Earliest claimed: 2012-05-09 - Assignees: Ncam Technologies Limited - Applicants: Ncam Technologies Limited - Inventors: Samuel Boivin; Brice Michoud - Web page: https://patentgrep.com/patent/US12217375B2 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06T 7/73: using feature-based methods - G06T 15/20: Perspective computation - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - H04N 5/2224: {related to virtual studio applications} - H04N 5/2621: {Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability} - H04N 5/265: Mixing - H04N 5/272: Means for inserting a foreground image in a background image, i.e. inlay, outlay - H04N 13/156: Mixing image signals - H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00) - H04N 13/275: from three-dimensional [3D] object models, e.g. computer-generated stereoscopic image signals - H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract The system generates real-time augmented reality video for TV broadcast, cinema or video games. The system includes a monoscopic video camera including a body, a stereoscopic video camera, and a processor. The system includes sensors, including multiple non-optical sensors, which provide real-time positioning data defining the 3D position and 3D orientation of the monoscopic video camera, or enable the 3D position and 3D orientation of the monoscopic video camera to be calculated. The processor is configured to use the real-time positioning data automatically to create, recall, render or modify computer generated 3D objects. The processor is configured to determine the 3D position and orientation of the monoscopic video camera with reference to a 3D map of the real-world generated whilst the camera is being used to capture video. The processor is configured to track the scene without a requirement for an initial or prior survey of the scene. ## Claims ### Claim 1 A markerless system, the system including: (i) a hand-held or portable monoscopic video camera, and lens encoders configured to output values including focus and iris data in real-time, the monoscopic video camera including, or in attachment with, the lens encoders; (ii) sensors including an accelerometer and a gyro sensing over six degrees of freedom; (iii) two witness cameras forming a stereoscopic system, in which the monoscopic video camera does not form part of the stereoscopic system; (iv) a camera tracking system in connection with the monoscopic video camera; and (v) a rendering station in connection with the camera tracking system; the markerless system being for mixing or compositing in real-time, computer generated 3D objects and a video feed from the video camera, to generate real-time augmented reality video for TV broadcast, cinema or video games, in which: (a) the sensors in or attached directly or indirectly to the video camera provide real-time positioning data defining the 3D position and 3D orientation of the video camera, or enabling the 3D position and 3D orientation of the video camera to be calculated, wherein the sensors are configured to output the real-time positioning data to the camera tracking system; (b) the two witness cameras forming the stereoscopic system are fixed directly or indirectly to the video camera; (c) the rendering station is configured to receive and to use the focus and iris data, and the real-time positioning data automatically to create, recall, render or modify computer generated 3D objects; (d) the rendering station is configured to mix-in or to composite the resulting computer generated 3D objects with the video feed from the video camera to provide augmented reality video for TV broadcast, cinema or video games; and in which: (e) the camera tracking system is configured to determine the 3D position and orientation of the video camera with reference to a 3D map of the real-world, wherein the camera tracking system is configured to generate the 3D map of the real-world, at least in part, by using the real-time 3D positioning data from the sensors plus a video flow in which the two witness cameras forming the stereoscopic system survey a scene, and in which the camera tracking system is configured to detect natural markers in the scene that have not been manually or artificially added to that scene. ### Claim 2 (depends on 1) The system of claim 1, wherein the values include zoom data, and wherein the rendering station is configured to receive and to use the zoom, focus and iris data, and the real-time positioning data automatically to create, recall, render or modify computer generated 3D objects. ### Claim 3 (depends on 1) The system of claim 1, the monoscopic video camera including an eyepiece, wherein the system is configured such that the mixed-in or composited computer generated 3D objects with the video feed from the video camera are streamed to the monoscopic video camera and are displayed by the eyepiece. ### Claim 4 (depends on 1) The system of claim 1, in which the sensors include a magnetometer measuring absolute heading in 3D, and hence the sensors sense over nine degrees of freedom. ### Claim 5 (depends on 1) The system of claim 1 in which the sensors include a 3D range sensor. ### Claim 6 (depends on 5) The system of claim 5 in which the depth of edges is refined by re-projecting 3D range sensor depths onto a high resolution video feed of the monoscopic video camera. ### Claim 7 (depends on 1) The system of claim 1 in which the pair of stereoscopic witness cameras each provides at least 100 fps, to enable the system to be fully initialized without a separate stage of purely surveying the scene to be tracked (called ‘instant survey’), and in which surveying takes place continuously whilst the camera is being used to capture video. ### Claim 8 (depends on 7) The system of claim 7 in which the pair of stereoscopic witness cameras provide images, and in which software executing on the camera tracking system processes the images and, even with the body not being moved at all, generates an instant 3D point cloud. ### Claim 9 (depends on 8) The system of claim 8 in which a depth of each pixel in the 3D point cloud is obtained using corresponding 2D texture patches obtained from each stereoscopic witness camera and an epi-polar line search algorithm. ### Claim 10 (depends on 1) The system of claim 1 in which the camera tracking system runs a fusion algorithm that combines optical flow data from the witness cameras with the real-time positioning data from the sensors. ### Claim 11 (depends on 10) The system of claim 10 in which the fusion algorithm is based on an Extended Kalman Filter prediction/correction technique to integrate outputs from, and to re-calibrate, all the sensors, selected from a group including an accelerometer, a gyroscope, a magnetometer, a 3D range sensor, to determine the position and orientation of the body. ### Claim 12 (depends on 11) The system of claim 11 in which the Extended Kalman Filter fusion algorithm uses confidence level data, associated with the output from each sensor, when determining how to merge the data from each sensor. ### Claim 13 (depends on 1) The system of claim 1 in which keyframes generated by the witness cameras are real-time images computed by the processor at four different resolution levels of the witness cameras' video feed. ### Claim 14 (depends on 5) The system of claim 5 in which the 3D range sensor is used to enhance the accuracy of a depth measurement associated with a reconstructed 3D point, or to reject that reconstructed 3D point. ### Claim 15 (depends on 14) The system of claim 14 in which the 3D range sensor is used for real-time depth keying to enable dynamic occlusion and suppress use of a green stage. ### Claim 16 (depends on 1) The system of claim 1, in which the camera tracking system is portable. ### Claim 17 (depends on 1) The system of claim 1 in which the camera tracking system is configured to generate a uniformly distributed point cloud in the 3D map. ### Claim 18 (depends on 1) The system of claim 1, in which real-time zoom, focus and iris settings of the video camera are measured, and used, together with the real-time positioning data, so that the 3D objects are correctly rendered in a desired location and orientation in a 3D scene. ### Claim 19 (depends on 1) The system of claim 1 that enables real-time, continuous surveying of the scene to generate a more complete point cloud defining the scene. ### Claim 20 (depends on 1) The system of claim 1 that attaches rotation invariant descriptors, to feature points detected in the scene to facilitate recovery of tracking. ### Claim 21 (depends on 1) The system of claim 1 that uses a constant velocity model associated with the information provided by the sensors to predict a next position of the monoscopic video camera using the previously correctly computed or confirmed position. ### Claim 22 (depends on 21) The system of claim 21 that uses that prediction to re-project a 3D point cloud onto the current frame, to enable a point matching algorithm to match points identified in the real-time video feed from the stereoscopic witness cameras and the projected points in the new, 3D point cloud. ### Claim 23 (depends on 22) The system of claim 22 that uses a Levenberg-Marquardt minimization scheme for camera tracking to minimize an error between the points identified in the real-time video feed from the stereoscopic witness cameras and the projected points in the new, 3D point cloud. ### Claim 24 (depends on 1) The system of claim 1 in which a user can use a 3D point cloud generated by the system to define 3D masks. ### Claim 25 (depends on 1) The system of claim 1 in which the monoscopic video camera and a witness camera are calibrated for frame acquisition delay using a modulated light source, such as by comparing light intensity curves associated with a flashing LED. ### Claim 26 (depends on 1) The system of claim 1 in which the rendering station is in wireless connection with the camera tracking system, and in which the rendering station is configured to receive wirelessly and to use the focus and iris data, and the real-time positioning data automatically to create, recall, render or modify computer generated 3D objects. ### Claim 27 A camera tracking system, the camera tracking system connectable to a hand-held or portable monoscopic video camera, the monoscopic video camera including, or in attachment with, lens encoders configured to output values including focus and iris data in real-time; wherein two witness cameras forming a stereoscopic system, in which the monoscopic video camera does not form part of the stereoscopic system, are fixed directly or indirectly to the video camera, wherein sensors including an accelerometer and a gyro sensing over six degrees of freedom are in or attached directly or indirectly to the video camera and provide real-time positioning data defining the 3D position and 3D orientation of the video camera, or enable the 3D position and 3D orientation of the video camera to be calculated, wherein the camera tracking system is configured to receive the focus and iris data, and the real-time positioning data from the sensors; wherein the camera tracking system is connectable to a rendering station, and is configured to transmit real-time positioning data, and the focus and iris data, to the rendering station; wherein the camera tracking system is configured to determine the 3D position and orientation of the video camera with reference to a 3D map of the real-world, wherein the camera tracking system is configured to generate the 3D map of the real-world, at least in part, by using the real-time 3D positioning data from the sensors plus a video flow in which the two witness cameras forming the stereoscopic system survey a scene, and in which the camera tracking system is configured to detect natural markers in the scene that have not been manually or artificially added to that scene. ### Claim 28 (depends on 27) The camera tracking system of claim 27, wherein the values include zoom data, and wherein the camera tracking system is configured to receive the zoom, focus and iris data, and the real-time positioning data from the sensors; wherein the camera tracking system is connectable to a rendering station, and is configured to transmit real-time positioning data, and the zoom, focus and iris data, to the rendering station. ### Claim 29 (depends on 27) The camera tracking system of claim 27, in which the pair of stereoscopic witness cameras provide images, and in which software executing on the camera tracking system processes the images and, even with the body not being moved at all, generates an instant 3D point cloud. ### Claim 30 (depends on 27) The camera tracking system of claim 27, in which the camera tracking system runs a fusion algorithm that combines optical flow data from the witness cameras with the real-time positioning data from the sensors. ### Claim 31 (depends on 27) The camera tracking system of claim 27, in which the camera tracking system is portable. ### Claim 32 (depends on 27) The camera tracking system of claim 27, in which the camera tracking system is configured to generate a uniformly distributed point cloud in the 3D map. ### Claim 33 (depends on 27) The camera tracking system of claim 27, that uses a constant velocity model associated with the information provided by the sensors to predict a next position of the monoscopic video camera using the previously correctly computed or confirmed position. ### Claim 34 (depends on 33) The camera tracking system of claim 33, that uses that prediction to re-project a 3D point cloud onto the current frame, to enable a point matching algorithm to match points identified in the real-time video feed from the stereoscopic witness cameras and the projected points in the new, 3D point cloud. ### Claim 35 (depends on 34) The camera tracking system of claim 34, that uses a Levenberg-Marquardt minimization scheme for camera tracking to minimize an error between the points identified in the real-time video feed from the stereoscopic witness cameras and the projected points in the new, 3D point cloud. ### Claim 36 (depends on 27) The camera tracking system of claim 27, wherein the system is configured to use that real-time positioning data automatically to create, recall, render or modify computer generated 3D objects; and the system is configured to mix-in or to composite the resulting computer generated 3D objects with the video feed from the video camera to provide augmented reality video for TV broadcast, cinema or video games. ### Claim 37 (depends on 27) The camera tracking system of claim 27, wherein the camera tracking system is wirelessly connectable to the rendering station, and is configured to wirelessly transmit real-time positioning data, and the focus and iris data, to the rendering station. ### Claim 38 A markerless method for mixing or compositing real-time, computer generated 3D objects and a video feed from a hand-held or portable monoscopic video camera, to generate augmented reality video for TV broadcast, cinema or video games, in which: (a) sensors, including an accelerometer and a gyro sensing over six degrees of freedom, in or attached directly or indirectly to the hand-held or portable video camera provide real-time positioning data defining the 3D position and 3D orientation of the video camera, or enabling the 3D position and 3D orientation of the video camera to be calculated, and the sensors output the real-time positioning data to a camera tracking system; (b) lens encoders, included in the monoscopic video camera, or in attachment with the monoscopic video camera, output values including focus and iris data in real-time to the camera tracking system; (c) two witness cameras forming a stereoscopic system are fixed directly or indirectly to the video camera; (d) a rendering station in connection with the camera tracking system receives and uses the focus and iris data, and the real-time positioning data to then automatically create, recall, render or modify computer generated 3D objects; (e) the rendering station mixes in or composites the resulting computer generated 3D objects with the video feed from the video camera to provide augmented reality video for TV broadcast, cinema or video games; and in which: (f) the camera tracking system determines the 3D position and orientation of the video camera with reference to a 3D map of the real-world, wherein the camera tracking system generates the 3D map of the real-world, at least in part, by using the real-time 3D positioning data from the sensors plus a video flow in which the two witness cameras forming the stereoscopic system survey a scene, and the camera tracking system detects natural markers in the scene that have not been manually or artificially added to that scene. ### Claim 39 (depends on 38) The method of claim 38, wherein the values include zoom data, and wherein the rendering station receives and uses the zoom, focus and iris data, and the real-time positioning data automatically to create, recall, render or modify computer generated 3D objects. ### Claim 40 (depends on 38) The method of claim 38 in which the computer generated 3D animation objects are automatically scaled to a correct size using stereoscopic algorithms. ### Claim 41 (depends on 38) The method of claim 38 in which the rendering station is in wireless connection with the camera tracking system.