Warped Perspective Correction

A method comprising:

• at a device having an image sensor, a display, one or more processors, and non-transitory memory;
• capturing, using the image sensor, an image of a physical environment;
• obtaining a plurality of initial depths respectively associated with a plurality of pixels of the image of the physical environment;
• generating a depth map for the image of the physical environment based on the plurality of initial depths and a respective plurality of confidences of the plurality of initial depths;
• transforming, using the one or more processors, the image of the physical environment based on the depth map and a difference between a perspective of the image sensor and a perspective of a user;
• and
• displaying, on the display, the transformed image.

A device comprising:

• an image sensor;
• a display;
• a non-transitory memory;
• and
• one or more processors to:
• capture, using the image sensor, an image of a physical environment;
• obtain a plurality of initial depths respectively associated with a plurality of pixels of the image of the physical environment;
• generate a depth map for the image of the physical environment based on the plurality of initial depths and a respective plurality of confidences of the plurality of initial depths;
• transform, using the one or more processors, the image of the physical environment based on the depth map and a difference between a perspective of the image sensor and a perspective of a user;
• and
• display, on the display, the transformed image.

A non-transitory memory storing one or more programs, which, when executed by one or more processors of a device with an image sensor and a display, cause the device to:

• capture, using the image sensor, an image of a physical environment;
• obtain a plurality of initial depths respectively associated with a plurality of pixels of the image of the physical environment;
• generate a depth map for the image of the physical environment based on the plurality of initial depths and a respective plurality of confidences of the plurality of initial depths;
• transform, using the one or more processors, the image of the physical environment based on the depth map and a difference between a perspective of the image sensor and a perspective of a user;
• and
• display, on the display, the transformed image.