Using 6dof pose information to align images from separated cameras

A head-mounted device (HMD) configured to align images generated by an integrated camera that is physically mounted to the HMD with images generated by a detached camera that is physically unmounted from the HMD, said HMD comprising:

• one or more processors;
• and
• one or more computer-readable hardware storage devices that store instructions that are executable by the one or more processors to cause the HMD to at least:
• generate a feature map of an environment in which both the integrated camera and the detached camera are operating;
• cause the integrated camera to generate a first image and the detached camera to generate a second image;
• use the feature map and simultaneous location and mapping (SLAM) to determine a pose of the integrated camera as represented within the first image and a pose of the detached camera as represented within the second image;
• access a depth map of the environment;
• and
• generate an overlaid image by reprojecting a perspective of the second image to align with a perspective of the first image and by overlaying at least a portion of the reprojected second image onto the first image, wherein (i) the pose of the integrated camera, (ii) the pose of the detached camera, and (iii) the depth map are used to perform said reprojecting.

A method for aligning images generated by an integrated camera that is physically mounted to a head-mounted device (HMD) with images generated by a detached camera that is physically unmounted from the HMD, said method comprising:

• generating a feature map of an environment in which both the integrated camera and the detached camera are operating;
• causing the integrated camera to generate a first image and the detached camera to generate a second image;
• using the feature map and simultaneous location and mapping (SLAM) to determine a pose of the integrated camera as represented within the first image and a pose of the detached camera as represented within the second image;
• accessing a depth map of the environment;
• generating an overlaid image by reprojecting a perspective of the second image to align with a perspective of the first image and by overlaying at least a portion of the reprojected second image onto the first image, wherein (i) the pose of the integrated camera, (ii) the pose of the detached camera, and (iii) the depth map are used to perform said reprojecting;
• and
• performing parallax correction on the overlaid image;
• and
• displaying the overlaid image on a display.