# US 2014/0192147 A1 · Automatic tracking matte system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0192147 A1 - Document ID: US20140192147A1 - Status: Published application - Kind: A1 - Application number: 14/209,403 - Filed: 2014-03-13 - Published: 2014-07-10 - Earliest claimed: 2011-12-01 - Assignees: Lightcraft Technology, LLC - Applicants: Lightcraft Technology, LLC - Inventors: NEWTON ELIOT MACK; PHILIP R. MASS - Web page: https://patentgrep.com/patent/US20140192147A1 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T 11/00: Two-dimensional [2D] image generation - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - H04N 5/265: Mixing - H04N 13/156: Mixing image signals - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A system for generating automatically tracking mattes that rapidly integrates live action and virtual composite images. ## Claims ### Claim 1 (canceled) ### Claim 2 (canceled) ### Claim 3 (canceled) ### Claim 4 (canceled) ### Claim 5 (canceled) ### Claim 6 (canceled) ### Claim 7 (canceled) ### Claim 8 (canceled) ### Claim 9 (canceled) ### Claim 10 (canceled) ### Claim 11 (canceled) ### Claim 12 (canceled) ### Claim 13 (canceled) ### Claim 14 (canceled) ### Claim 15 (canceled) ### Claim 16 (canceled) ### Claim 17 (canceled) ### Claim 18 (canceled) ### Claim 19 (canceled) ### Claim 20 (canceled) ### Claim 21 A system comprising: a video processor configured to convert a matte shape to 3D spatial coordinates and redrawn with lens and camera data matching the live action camera and lens and to use the matte shape to selectively process portions of a live action image in combination with at least one other image to generate a composited image. ### Claim 22 (depends on 21) The system of claim 21 wherein the matte shape is defined by matte points lying in a plane normal to the line of sight of the live action camera. ### Claim 23 (depends on 22) The system of claim 22 wherein the matte points are manually-selected matte points. ### Claim 24 (depends on 21) The system of claim 21 wherein the matte shape is defined at least in part by a recording of the original camera location, focal length and distance between the camera and the plane of the matte at the time of matte creation. ### Claim 25 (depends on 21) The system of claim 21 wherein the matte shape is defined by a scaling of the matte shape proportionally to the original focal length of the camera and along the original camera axis as the user adjusts the matte distance after the matte has been created. ### Claim 26 (depends on 21) The system of claim 21 wherein the matte shape is defined by a translating the scene camera by the user, and then an adjusting of the distance of the matte plane from the original camera location until the matte points line up with their respective scene elements in the new camera position, thereby resolving the position of the matte shape with the object being outlined in 3D space. ### Claim 27 (depends on 21) The system of claim 21 wherein the matte shape is defined by a constraining of the drawn points so that the points can be moved closer or further away from the original camera location, but only along the optical projection axis of the original camera and constrained to a plane normal to that of the original camera axis at time of the matte was originally created. ### Claim 28 (depends on 21) The system of claim 21 wherein the user sets the matte distance by directly measuring the distance between the live action camera and the live action object that is to be matted at the time of matte creation. ### Claim 29 (depends on 21) The system of claim 21 wherein the at least one other image is a virtual background created with the same camera perspective as the live action foreground. ### Claim 30 A method comprising: selecting points on a computer screen to generate a rough outline around an object to be removed or preserved from a live action image from a live action camera; the selecting including beginning or ending the outline at an edge of the screen; the beginning or ending causing the matte to be automatically extended a predetermined distance out from the edge of the screen without requiring the camera to move back and forth; and the matte being comprised of three-dimensional geometry in an arbitrary three-dimensional plane, so that areas extended past the geometry visible to the user are projected into three-dimensional space. ### Claim 31 (depends on 30) The method of claim 30 wherein the matte lies in a plane perpendicular to an optical axis of the live action camera. ### Claim 32 (depends on 31) The method of claim 31 wherein the plane can be adjusted further or closer to the original live action camera location at time of matte creation. ### Claim 33 (depends on 31) The method of claim 31 wherein the selected points are scaled along the original camera's axis and focal length projection as the user changes the matte plane distance. ### Claim 34 (depends on 31) The method of claim 31 wherein the selected points are viewed from the new camera position with correct distortion of the lens in the new position. ### Claim 35 (depends on 31) The method of claim 31 wherein the matte can be scaled closer or nearer to the original camera location even after the scene camera has been moved, so that the user can triangulate the 3D position of the matte in space by aligning the matte to live action features in two camera locations. ### Claim 36 (depends on 31) The method of claim 31 wherein the selecting includes restricting the 2D dragging movement of the matte points within a plane normal to the original camera axis. ### Claim 37 A method comprising: generating a 2D outline around an object to be removed or preserved in a 2D display of a live action image from a live action camera; converting the 2D outline to a 3D mesh on a plane normal to the optical axis of the original live action camera location; and projecting the 3D mesh from a virtual camera having the same position and orientation as the live action camera to line up the 3D mesh with elements of the live action image. ### Claim 38 (depends on 37) The method of claim 37 further comprising the 3D mesh at least approximately tracking the live action object as the live action camera is moved to a new position relative to the object, and the 3D mesh scaled relative to the original camera position, orientation and focal length until 3D mesh aligns with the live action object from the new position, orientation and focal length of the live action camera. ### Claim 39 (depends on 38) The method of claim 38 wherein the tracking includes adjusting the location of the points of the outline in three dimensional space so that the 3D mesh scales as the distance from the original camera location changes. ### Claim 40 (depends on 37) The method of claim 37 wherein the 2D outline lies in a plane perpendicular to the line of sight of the live action camera. ### Claim 41 (depends on 37) The method of claim 37 wherein the generating includes manually adjusting the distance between the original camera position and the normal plane on which the selected matte mesh points are located. ### Claim 42 (depends on 37) The method of claim 37 wherein the projecting includes moving the live action camera to a new location, and adjusting the distance between the matte plane and the original camera until the matte points line up with the live action object in the new live action camera position and orientation, thereby locating the matte object in 3D space. ### Claim 43 (depends on 37) The method of claim 37 further comprising after the projecting the 3D mesh, manipulating the points in a 2D interface while constraining the points to move on a plane normal to the camera axis in which the points were originally created. ### Claim 44 A method comprising: manipulating a 3D object via a 2D interface by forcing 3D points to lie in a plane and calculating their 3D positions by dragging around a 2D dot on a 2D interface; wherein the plane is perpendicular to the tracked camera position from where the points were original drawn. ### Claim 45 (depends on 44) The method of claim 44 wherein the distance between the matte plane and the scene camera location is set by measuring the distance between the live action camera and the live action object by a measuring device such as a ruler or an electronic rangefinder. ### Claim 46 (depends on 44) The method of claim 44 wherein the manipulating includes tracking at least approximately the 3D object by locating the 3D matte at approximately the same position as the actual live action object via direct measurement of the distance between the camera and the 3D object at time of the matte creation, or triangulating the distance to the 3D object by translating the scene camera and adjusting the distance between the matte object and the original camera until the matte object lines up with the live action object in the new camera position.