Gaze and head tracking informed hardware-based passthrough noise reduction

An electronic device comprising:

• a non-transitory computer-readable storage medium;
• and
• one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the electronic device to perform operations comprising:
• obtaining image content from an image sensor, the image content comprising frames corresponding to views of a three-dimensional (3D) environment, the frames comprising a current frame and one or more previously-captured frames;
• predicting a user head pose based on sensor data obtained via one or more sensors, the user head pose corresponding to a viewpoint of the image sensor with respect to the 3D environment during capture of the current frame;
• estimating depth attributes corresponding to distances of one or more elements of a scene depicted in the current frame;
• generating an enhanced frame by combining the current frame with the one or more previously-captured frames, wherein the combining is based on aligning at least one of the previously-captured frames with the current frame based on the predicted user head pose and the estimated depth attributes;
• and
• presenting, to the user via one or more displays of the electronic device, the enhanced frame.

A non-transitory computer-readable storage medium storing program instructions executable via one or more processors, of an electronic device, to perform operations comprising:

• obtaining image content from an image sensor, the image content comprising frames corresponding to views of a three-dimensional (3D) environment, the frames comprising a current frame and one or more previously-captured frames;
• predicting a user head pose based on sensor data obtained via one or more sensors, the user head pose corresponding to a viewpoint of the image sensor with respect to the 3D environment during capture of the current frame;
• estimating depth attributes corresponding to distances of one or more elements of a scene depicted in the current frame;
• generating an enhanced frame by combining the current frame with the one or more previously-captured frames, wherein the combining is based on aligning at least one of the previously-captured frames with the current frame based on the predicted user head pose and the estimated depth attributes;
• and
• presenting, to the user via one or more displays of the electronic device, the enhanced frame.