Mixed reality video production with detached camera

A system for producing a mixed reality video comprising:

• a physically detached camera;
• a background set;
• a display device;
• and
• a processor configured to:
• create a virtual reality scene;
• detect a wireless connection of the camera to the processor;
• guide a subject experiencing the virtual scene through a calibration process that results in a determination of a transformation matrix from a frame of reference of the camera to a frame of reference of the virtual reality scene;
• receive a video frame of the subject and the background set captured by the camera;
• receive a position and orientation of the camera in the camera's frame of reference corresponding to when the camera captured the video frame;
• determine a position and orientation of the camera in the frame of reference of the virtual scene;
• create a view of the virtual scene from a perspective of the position and orientation of the camera in the frame of reference of the virtual scene;
• remove the background from the video frame;
• composit the video frame of the subject with the view of the virtual scene;
• and
• relay the composited view of the virtual scene to the display device.

A non-transitory computer readable medium comprising computer-readable instructions, which, when executed by a processor cause the processor to:

• create a virtual reality scene;
• detect a wireless connection of a detached camera to the processor;
• guide a subject experiencing the virtual scene through a calibration process that results in a determination of a transformation matrix from a frame of reference of the camera to a frame of reference of the virtual reality scene;
• receive a video frame of the subject and a background to the subject captured by the camera;
• receive a position and orientation of the camera in the camera's frame of reference corresponding to when the camera captured the video frame;
• determine a position and orientation of the camera in the frame of reference of the virtual scene;
• create a view of the virtual scene from a perspective of the position and orientation of the camera in the frame of reference of the virtual scene;
• remove the background from the video frame;
• composit the video frame of the subject with the view of the virtual scene;
• and
• relay the composited view of the virtual scene to the display device.