Photogrammetric alignment for immersive content production

An immersive content production system comprising:

• a performance area;
• one or more displays surrounding the performance area for displaying virtual content;
• a taking camera for capturing images of the performance area and a portion of the one or more displays displaying the virtual content;
• one or more sensors for detecting a current position and a current orientation of the taking camera in a taking camera coordinate system;
• a computer memory storing:
• a survey of the performance area associated with a survey coordinate system having an origin corresponding to a reference point in the performance area, the survey including a point cloud representing a first physical object in the performance area;
• and
• the virtual content associated with a virtual coordinate system, the virtual content including a first virtual object corresponding to the first physical object;
• and
• one or more computer processors configured to:
• in a survey graph hierarchy, constrain the point cloud and the taking camera coordinate system as child nodes of the origin of the survey coordinate system, so that the point cloud and the taking camera coordinate system inherit changes to the origin of the survey coordinate system;
• apply a transformation to the origin of the survey coordinate system so that at least a portion of the point cloud that represents the first physical object is substantially aligned with a portion of the virtual content that represents the first virtual object;
• and
• cause the virtual content to be displayed on the one or more displays from a perspective of the taking camera at the current position and the current orientation while the taking camera is capturing the images, the images including the first physical object and at least a portion of the one or more displays on which the first virtual object is being displayed.