# US 2021/0342971 A1 · Photogrammetric alignment for immersive content production

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342971 A1
- Document ID: US20210342971A1
- Status: Published application
- Kind: A1
- Application number: 17/229,444
- Filed: 2021-04-13
- Published: 2021-11-04
- Earliest claimed: 2020-04-29
- Assignees: Lucasfilm Entertainment Company Ltd.
- Applicants: Lucasfilm Entertainment Company Ltd.
- Inventors: Douglas G. Watkins; Paige M. Warner; Dacklin R. Young
- Web page: https://patentgrep.com/patent/US20210342971A1

## CPC

- G06T 3/147: using affine transformations
- G06T 7/33: using feature-based methods
- G06T 7/73: using feature-based methods
- G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method of content production includes generating a survey of a performance area that includes a point cloud representing a first physical object, in a survey graph hierarchy, constraining the point cloud and a taking camera coordinate system as child nodes of an origin of a survey coordinate system, obtaining virtual content including a first virtual object that corresponds to the first physical object, applying a transformation to the origin of the survey coordinate system so that at least a portion of the point cloud that represents the first physical object is substantially aligned with a portion of the virtual content that represents the first virtual object, displaying the first virtual object on one or more displays from a perspective of the taking camera, capturing, using the taking camera, one or more images of the performance area, and generating content based on the one or more images.

## Claims

### Claim 1

A method of content production, the method comprising:
generating a survey of a performance area, the survey including a point cloud representing one or more physical objects including a first physical object in the performance area, the point cloud being associated with a survey coordinate system having an origin corresponding to a reference point in the performance area;
in a survey graph hierarchy, constraining the point cloud and a taking camera coordinate system as child nodes of the origin of the survey coordinate system, so that the point cloud and the taking camera coordinate system inherit changes to the origin of the survey coordinate system;
obtaining virtual content including one or more virtual objects associated with a virtual coordinate system, the one or more virtual objects including a first virtual object that corresponds to the first physical object;
applying a transformation to the origin of the survey coordinate system so that at least a portion of the point cloud that represents the first physical object is substantially aligned with a portion of the virtual content that represents the first virtual object;
detecting a current position and a current orientation of the taking camera in the taking camera coordinate system;
displaying the first virtual object on one or more displays surrounding the performance area from a perspective of the taking camera at the current position and the current orientation;
capturing, using the taking camera, one or more images of the performance area including the first physical object and at least a portion of the one or more displays on which the first virtual object is being displayed; and
generating content based on the one or more images.

### Claim 2 (depends on 1)

The method of claim 1, further comprising acquiring, using one or more alignment devices positioned around the performance area, positional information of the one or more physical objects in the performance area, and wherein generating the survey of the performance area is based on the positional information.

### Claim 3 (depends on 2)

The method of claim 2, wherein:
the one or more alignment devices comprise a plurality of alignment cameras positioned at a plurality of locations around the performance area;
the positional information comprises a plurality of 2-D images captured by the plurality of alignment cameras; and
generating the survey comprises generating the point cloud based on the plurality of 2-D images by using a photogrammetric technique.

### Claim 4 (depends on 2)

The method of claim 2, wherein the one or more alignment devices comprise one or more 3-D sensors.

### Claim 5 (depends on 1)

The method of claim 1, wherein the virtual content comprise a 3-D mesh or a point cloud representing the one or more virtual objects.

### Claim 6 (depends on 1)

The method of claim 1, further comprising determining the transformation by:
identifying a plurality of landmark nulls on the point cloud representing the first physical object;
identifying a plurality of target nulls on the first virtual object, each respective target null corresponding to a respective landmark null on the point cloud; and
determining the transformation so that, after applying the transformation to the origin of the survey coordinate system, the plurality of landmark nulls on the point cloud substantially matches with the plurality of target nulls.

### Claim 7 (depends on 6)

The method of claim 6, wherein determining the transformation is performed using a least square regression algorithm.

### Claim 8 (depends on 6)

The method of claim 6, further comprising constraining the plurality of landmark nulls as a child node of the origin of the survey coordinate system in the survey graph hierarchy.

### Claim 9 (depends on 1)

The method of claim 1, further comprising determining the transformation by performing registration between the portion of the point cloud and the portion of the virtual content using a best fit algorithm.

### Claim 10 (depends on 9)

The method of claim 9, wherein the best fit algorithm comprises an iterative closest point (ICP) algorithm.

### Claim 11 (depends on 1)

The method of claim 1, wherein the transformation includes one or more of a rotation, a translation, and a scaling.

### Claim 12 (depends on 1)

The method of claim 1, wherein determining the current position and the current orientation of the taking camera is performed by using one or more motion sensors, one or more alignment devices, one or more GPS receivers attached to the taking camera, or a combination thereof.

### Claim 13 (depends on 1)

The method of claim 1, wherein the one or more images comprise one or more frames of a video or a movie.

### Claim 14

An immersive content production system comprising:
a performance area;
one or more displays surrounding the performance area for displaying virtual content;
a taking camera for capturing images of the performance area and a portion of the one or more displays displaying the virtual content;
one or more sensors for detecting a current position and a current orientation of the taking camera in a taking camera coordinate system;
a computer memory storing:
a survey of the performance area associated with a survey coordinate system having an origin corresponding to a reference point in the performance area, the survey including a point cloud representing a first physical object in the performance area; and
the virtual content associated with a virtual coordinate system, the virtual content including a first virtual object corresponding to the first physical object; and
one or more computer processors configured to:
in a survey graph hierarchy, constrain the point cloud and the taking camera coordinate system as child nodes of the origin of the survey coordinate system, so that the point cloud and the taking camera coordinate system inherit changes to the origin of the survey coordinate system;
apply a transformation to the origin of the survey coordinate system so that at least a portion of the point cloud that represents the first physical object is substantially aligned with a portion of the virtual content that represents the first virtual object; and
cause the virtual content to be displayed on the one or more displays from a perspective of the taking camera at the current position and the current orientation while the taking camera is capturing the images, the images including the first physical object and at least a portion of the one or more displays on which the first virtual object is being displayed.

### Claim 15 (depends on 14)

The immersive content production system of claim 14, further comprising one or more alignment devices positioned around the performance area and configured to acquire positional information of one or more physical objects in the performance area, and wherein the one or more computer processors are further configured to generate the survey of the performance area based on the positional information.

### Claim 16 (depends on 15)

The immersive content production system of claim 15, wherein:
the one or more alignment devices comprise a plurality of alignment cameras positioned at a plurality of locations around the performance area;
the positional information comprises a plurality of 2-D images captured by the plurality of alignment cameras; and
generating the survey comprises generating the point cloud based on the plurality of 2-D images by using a photogrammetric technique.

### Claim 17 (depends on 15)

The immersive content production system of claim 15, wherein the one or more alignment devices comprise one or more 3-D sensors.

### Claim 18 (depends on 14)

The immersive content production system of claim 14, wherein the one or more computer processors are further configured to determine the transformation by:
identifying a plurality of landmark nulls on the point cloud representing the first physical object;
identifying a plurality of target nulls on the first virtual object, each respective target null corresponding to a respective landmark null on the point cloud; and
determining the transformation so that, after applying the transformation to the origin of the survey coordinate system, the plurality of landmark nulls on the point cloud substantially matches with the plurality of target nulls.

### Claim 19 (depends on 18)

The immersive content production system of claim 18, wherein the one or more computer processors are further configured to:
determine the transformation by performing registration between the portion of the point cloud that represents the first physical object and a portion of the virtual content that represents the first virtual object using a best fit algorithm.

### Claim 20 (depends on 14)

The immersive content production system of claim 14, wherein the transformation includes one or more of a rotation, a translation, and a scaling.
