Advanced temporal low light filtering with global and local motion compensation

A computer system configured to generate a temporally filtered image that is designed to compensate for global motions of a camera and to compensate for local motions of one or more objects that are locally moving within a scene, said computer system comprising:

• one or more processors;
• and
• one or more computer-readable hardware storage devices that store instructions that are executable by the one or more processors to cause the computer system to:
• acquire a history frame and a current frame of a scene, wherein the current frame is generated by a camera;
• in response to determining a first pose of the camera when the current frame was generated, perform a global motion compensation operation on the history frame to reproject a second pose of the history frame to match the first pose;
• after performing the global motion compensation operation, compare the motion compensated history frame against the current frame to identify a set of pixels that are located at a first set of positions within the motion compensated history frame but that are located at a second set of positions within the current frame, the set of pixels corresponding to one or more objects that moved locally within the scene, the second set of positions being different than the first set of positions;
• for each pixel in the set of pixels, compute a corresponding optical flow vector such that a plurality of optical flow vectors is computed, wherein the plurality of optical flow vectors map movement of the one or more objects, where the movement is represented by the set of pixels being at the first set of positions in the motion compensated history frame and being at the second set of positions in the current frame;
• generate an adjusted history frame from the motion compensated history frame by applying the plurality of optical flow vectors to the set of pixels included in the motion compensated history frame to shift the set of pixels from being at the first set of positions in the motion compensated history frame to being at new positions such that the adjusted history frame reflects the new positions of the set of pixels, said new positions in the adjusted history frame corresponding to the second set of positions that were identified in the current frame for the set of pixels;
• and
• after shifting the set of pixels to the new positions in the adjusted history frame, temporally filter the current frame with the adjusted history frame to generate a temporally filtered image.

A computer system configured to generate a temporally filtered image that is designed to compensate for global motions of a camera and to compensate for local motions of one or more objects that are locally moving within a scene, said computer system comprising:

• one or more processors;
• and
• one or more computer-readable hardware storage devices that store instructions that are executable by the one or more processors to cause the computer system to:
• acquire a history frame and a current frame of a scene, wherein the current frame is generated by a camera;
• acquire inertial measurement unit (IMU) data that reflects a movement of the camera between a time when the history frame is acquired and a time when the current frame is generated by the camera;
• in response using the IMU data to determine a first pose of the camera when the current frame was generated, perform a global motion compensation operation on the history frame to reproject a second pose of the history frame to match the first pose;
• after performing the global motion compensation operation, compare the motion compensated history frame against the current frame to identify a set of pixels that are located at a first set of positions within the motion compensated history frame but that are located at a second set of positions within the current frame, the set of pixels corresponding to one or more objects that moved locally within the scene, the second set of positions being different than the first set of positions;
• for each pixel in the set of pixels, compute a corresponding optical flow vector such that a plurality of optical flow vectors is computed, wherein the plurality of optical flow vectors map movement of the one or more objects, where the movement is represented by the set of pixels being at the first set of positions in the motion compensated history frame and being at the second set of positions in the current frame;
• generate an adjusted history frame from the motion compensated history frame by applying the plurality of optical flow vectors to the set of pixels included in the motion compensated history frame to shift the set of pixels from being at the first set of positions in the motion compensated history frame to being at new positions such that the adjusted history frame reflects the new positions of the set of pixels, said new positions in the adjusted history frame corresponding to the second set of positions that were identified in the current frame for the set of pixels;
• and
• after shifting the set of pixels to the new positions in the adjusted history frame, temporally filter the current frame with the adjusted history frame to generate a temporally filtered image.