System for mixing or compositing in real-time, computer generated 3D objects and a video feed from a film camera

A camera tracking system, the camera tracking system connectable to a hand-held or portable monoscopic video camera, the monoscopic video camera including, or in attachment with, lens encoders configured to output values including focus and iris data in real-time;

• wherein two witness cameras forming a stereoscopic system, in which the monoscopic video camera does not form part of the stereoscopic system, are fixed directly or indirectly to the video camera, wherein sensors including an accelerometer and a gyro sensing over six degrees of freedom are in or attached directly or indirectly to the video camera and provide real-time positioning data defining the 3D position and 3D orientation of the video camera, or enable the 3D position and 3D orientation of the video camera to be calculated, wherein the camera tracking system is configured to receive the focus and iris data, and the real-time positioning data from the sensors;
• wherein the camera tracking system is connectable to a rendering station, and is configured to transmit real-time positioning data, and the focus and iris data, to the rendering station;
• wherein the camera tracking system is configured to determine the 3D position and orientation of the video camera with reference to a 3D map of the real-world, wherein the camera tracking system is configured to generate the 3D map of the real-world, at least in part, by using the real-time 3D positioning data from the sensors plus a video flow in which the two witness cameras forming the stereoscopic system survey a scene, and in which the camera tracking system is configured to detect natural markers in the scene that have not been manually or artificially added to that scene.

A markerless method for mixing or compositing real-time, computer generated 3D objects and a video feed from a hand-held or portable monoscopic video camera, to generate augmented reality video for TV broadcast, cinema or video games, in which:

• (a) sensors, including an accelerometer and a gyro sensing over six degrees of freedom, in or attached directly or indirectly to the hand-held or portable video camera provide real-time positioning data defining the 3D position and 3D orientation of the video camera, or enabling the 3D position and 3D orientation of the video camera to be calculated, and the sensors output the real-time positioning data to a camera tracking system;
• (b) lens encoders, included in the monoscopic video camera, or in attachment with the monoscopic video camera, output values including focus and iris data in real-time to the camera tracking system;
• (c) two witness cameras forming a stereoscopic system are fixed directly or indirectly to the video camera;
• (d) a rendering station in connection with the camera tracking system receives and uses the focus and iris data, and the real-time positioning data to then automatically create, recall, render or modify computer generated 3D objects;
• (e) the rendering station mixes in or composites the resulting computer generated 3D objects with the video feed from the video camera to provide augmented reality video for TV broadcast, cinema or video games;
• and in which:
• (f) the camera tracking system determines the 3D position and orientation of the video camera with reference to a 3D map of the real-world, wherein the camera tracking system generates the 3D map of the real-world, at least in part, by using the real-time 3D positioning data from the sensors plus a video flow in which the two witness cameras forming the stereoscopic system survey a scene, and the camera tracking system detects natural markers in the scene that have not been manually or artificially added to that scene.