IPD correction and reprojection for accurate mixed reality object placement

One or more hardware storage devices having stored thereon computer-executable instructions that are executable by a computer system to cause the computer system to:

• determine an interpupil distance of a user;
• cause a stereo camera pair, which includes a first camera and a second camera pointing in non-parallel directions, to capture images of an environment, the first camera being used to capture a first image of the environment and the second camera being used to capture a second image of the environment;
• generate a partial depth map based on pixel disparities identified between pixels of the first image and pixels of the second image, wherein the partial depth map is designed to be limited as a result of the partial depth map selectively including depth values for only a limited set of image pixels that are common between the first image and the second image such that the partial depth map omits depth values for some common image pixels commonly represented in both the first image and the second image;
• after filtering some of the depth values from the partial depth map to remove noise, insert new depth values for those depth values that were filtered such that the partial depth map remains limited despite the new depth values being inserted into the partial depth map;
• apply epipolar transforms to the first and second images to generate a transformed first image and a transformed second image, a center-line perspective of the transformed first image being parallel to a center-line perspective of the transformed second image as a result of the epipolar transforms;
• and
• generate a first passthrough visualization and a second passthrough visualization by reprojecting, using the partial depth map, the transformed first image and the transformed second image, the reprojecting causing the transformed first image's center-line perspective to be in alignment with a left pupil of the user and the transformed second image's center-line perspective to be in alignment with a right pupil of the user.

A method for generating passthrough visualizations, said method being implemented by one or more processors of a computer system and comprising:

• determining an interpupil distance of a user;
• causing a stereo camera pair, which includes a first camera and a second camera, to capture images of an environment, the first camera being used to capture a first image of the environment and the second camera being used to capture a second image of the environment;
• generating a partial depth map based on pixel disparities identified between pixels of the first image and pixels of the second image, wherein the partial depth map is designed to be limited as a result of the partial depth map selectively including depth values for only a limited set of image pixels that are common between the first image and the second image such that the partial depth map omits depth values for some common image pixels commonly represented in both the first image and the second image;
• after filtering some of the depth values from the partial depth map to remove noise, inserting new depth values for those depth values that were filtered such that the partial depth map remains limited despite the new depth values being inserted into the partial depth map;
• applying epipolar transforms to the first and second images to generate a transformed first image and a transformed second image, a center-line perspective of the transformed first image being parallel to a center-line perspective of the transformed second image as a result of the epipolar transforms;
• and
• generating a first passthrough visualization and a second passthrough visualization by reprojecting, using the partial depth map, the transformed first image and the transformed second image, the reprojecting causing the transformed first image's center-line perspective to be in alignment with a left pupil of the user and the transformed second image's center-line perspective to be in alignment with a right pupil of the user.