Ipd correction and reprojection for accurate mixed reality object placement

One or more hardware storage devices having stored thereon computer-executable instructions, the computer-executable instructions being executable by a computer system that includes a head-mounted device to cause the computer system to reconstruct a perspective captured in an image so that the captured perspective matches a perspective of a user who is wearing the head-mounted device by causing the computer system to:

• determine an interpupil distance of the user who is wearing the head-mounted device, the head-mounted device including a stereo camera pair, the stereo camera pair including a left camera and a right camera, the stereo camera pair being used to capture images of a surrounding environment, wherein a center-line perspective of an image captured by the left camera has a non-parallel alignment with respect to a center-line perspective of an image captured by the right camera;
• use the left camera to capture a raw left image of the surrounding environment and use the right camera to capture a raw right image of the surrounding environment;
• apply camera distortion corrections to the raw left and right images to generate a corrected left image and a corrected right image, the corrected left image having a same center-line perspective as the raw left image, the corrected right image having a same center-line perspective as the raw right image;
• apply epipolar transforms to the corrected left and right images to generate a transformed left image and a transformed right image, wherein a center-line perspective of the transformed left image is parallel to a center-line perspective of the transformed right image as a result of the epipolar transforms;
• generate a depth map that is based on a combination of the transformed left image and the transformed right image;
• and
• generate a left passthrough visualization and a right passthrough visualization by reprojecting the transformed left image and the transformed right image, the reprojecting being performed using a result obtained from the depth map, the reprojecting causing the transformed left image's center-line perspective to be in alignment with a left pupil of the user and the transformed right image's center-line perspective to be in alignment with a right pupil of the user.

A method for reconstructing a perspective captured in an image so that the captured perspective matches a perspective of a user who is wearing a head-mounted device, the method being implemented by one or more processors of the head-mounted device, the method comprising:

• determining an interpupil distance of the user who is wearing the head-mounted device, the head-mounted device including a stereo camera pair, the stereo camera pair including a left camera and a right camera, the stereo camera pair being used to capture images of a surrounding environment, wherein a center-line perspective of an image captured by the left camera has a non-parallel alignment with respect to a center-line perspective of an image captured by the right camera;
• using the left camera to capture a raw left image of the surrounding environment and using the right camera to capture a raw right image of the surrounding environment;
• applying camera distortion corrections to the raw left and right images to generate a corrected left image and a corrected right image, the corrected left image having a same center-line perspective as the raw left image, the corrected right image having a same center-line perspective as the raw right image;
• applying epipolar transforms to the corrected left and right images to generate a transformed left image and a transformed right image, wherein a center-line perspective of the transformed left image is parallel to a center-line perspective of the transformed right image as a result of the epipolar transforms;
• generating a depth map that is based on a combination of the transformed left image and the transformed right image;
• and
• generating a left passthrough visualization and a right passthrough visualization by reprojecting the transformed left image and the transformed right image, the reprojecting being performed using a result obtained from the depth map, the reprojecting causing the transformed left image's center-line perspective to be in alignment with a left pupil of the user and the transformed right image's center-line perspective to be in alignment with a right pupil of the user.