# US 2023/0274401 A1 · Advanced temporal low light filtering with global and local motion compensation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0274401 A1
- Document ID: US20230274401A1
- Status: Published application
- Kind: A1
- Application number: 17/682,315
- Filed: 2022-02-28
- Published: 2023-08-31
- Earliest claimed: 2022-02-28
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Raymond Kirk PRICE; Michael BLEYER; Christopher Douglas EDMONDS
- Web page: https://patentgrep.com/patent/US20230274401A1

## CPC

- G06T 5/50: using two or more images, e.g. averaging or subtraction
- H04N 5/145: {Movement estimation (for video coding H04N19/51)}
- G06T 3/00: Geometric image transformations in the plane of the image
- G06T 5/70: Denoising; Smoothing
- G06T 7/223: using block-matching
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06V 10/30: Noise filtering
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- H04N 23/6811: {based on the image signal}
- H04N 23/6812: {based on additional sensors, e.g. acceleration sensors}
- H04N 23/81: for suppressing or minimising disturbance in the image signal generation
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20216: Image averaging
- G06T 2207/30196: Human being; Person
- G06T 2207/30221: Sports video; Sports image
- G06T 2207/30241: Trajectory
- G06T 2207/30244: Camera pose
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Techniques for generating a temporally filtered image designed to compensate for global motions of a camera and to compensate for local motions of an object are disclosed. A history frame and a current frame are acquired. A global motion compensation operation is performed on the history frame to reproject a pose of the history frame to match the pose of the current frame. The history frame is compared against the current frame to identify pixels that represent moving objects. For each of those pixels, an optical flow vector is computed. The optical flow vectors are then applied to those pixels to shift those pixels to new locations. These new positions, which are in the history frame, correspond to positions that were identified in the current frame. Afterwards, the current frame is temporally filtered with the history frame.

## Claims

### Claim 1

A method for generating a temporally filtered image that is designed to compensate for global motions of a camera and to compensate for local motions of one or more objects that are locally moving within a scene, said method comprising:
acquiring a history frame and a current frame of a scene, wherein the current frame is generated by a camera;
in response to determining a first pose of the camera when the current frame was generated, performing a global motion compensation operation on the history frame to reproject a second pose of the history frame to match the first pose;
after performing the global motion compensation operation, comparing the motion compensated history frame against the current frame to identify a set of pixels that are located at a first set of positions within the motion compensated history frame but that are located at a second set of positions within the current frame, the set of pixels corresponding to one or more objects that moved locally within the scene, the second set of positions being different than the first set of positions;
for each pixel in the set of pixels, computing a corresponding optical flow vector such that a plurality of optical flow vectors is computed, wherein the plurality of optical flow vectors map movement of the one or more objects, where the movement is represented by the set of pixels being at the first set of positions in the motion compensated history frame and being at the second set of positions in the current frame;
generating an adjusted history frame from the motion compensated history frame by applying the plurality of optical flow vectors to the set of pixels included in the motion compensated history frame to shift the set of pixels from being at the first set of positions in the motion compensated history frame to being at new positions such that the adjusted history frame reflects the new positions of the set of pixels, said new positions in the adjusted history frame corresponding to the second set of positions that were identified in the current frame for the set of pixels; and
after shifting the set of pixels to the new positions in the adjusted history frame, temporally filtering the current frame with the adjusted history frame to generate a temporally filtered image.

### Claim 2 (depends on 1)

The method of claim 1, wherein the temporally filtered image operates as a new history frame for a subsequent iteration of said method.

### Claim 3 (depends on 1)

The method of claim 1, wherein the history frame is a previously generated temporally filtered image.

### Claim 4 (depends on 1)

The method of claim 1, wherein inertial measurement unit (IMU) data is used to perform the global motion compensation operation.

### Claim 5 (depends on 1)

The method of claim 1, wherein comparing the motion compensated history frame against the current frame to identify the set of pixels is performed after one or both of the current frame and the motion compensated history frame are down-sampled to a reduced resolution.

### Claim 6 (depends on 1)

The method of claim 1, wherein the camera is one of a single-photon avalanche diode (SPAD) camera, a single photon detector, a super cooled camera, a scientific camera module, or an image intensifier II plus a complementary metal-oxide semiconductor (CMOS) camera.

### Claim 7 (depends on 1)

The method of claim 1, wherein the camera is a low noise camera.

### Claim 8 (depends on 1)

The method of claim 1, wherein temporally filtering the current frame with the adjusted history frame to generate the temporally filtered image includes averaging pixels in the adjusted history frame with pixels in the current frame.

### Claim 9 (depends on 1)

The method of claim 1, wherein comparing the motion compensated history frame against the current frame to identify the set of pixels includes computing an absolute difference between the motion compensated history frame and the current frame.

### Claim 10 (depends on 1)

The method of claim 1, wherein, for content that was previously occluded in the motion compensated history frame by the set of pixels but that is now not occluded by the set of pixels in the adjusted history frame, image data for said content is obtained from the current frame.

### Claim 11

A computer system configured to generate a temporally filtered image that is designed to compensate for global motions of a camera and to compensate for local motions of one or more objects that are locally moving within a scene, said computer system comprising:
one or more processors; and
one or more computer-readable hardware storage devices that store instructions that are executable by the one or more processors to cause the computer system to:
acquire a history frame and a current frame of a scene, wherein the current frame is generated by a camera;
in response to determining a first pose of the camera when the current frame was generated, perform a global motion compensation operation on the history frame to reproject a second pose of the history frame to match the first pose;
after performing the global motion compensation operation, compare the motion compensated history frame against the current frame to identify a set of pixels that are located at a first set of positions within the motion compensated history frame but that are located at a second set of positions within the current frame, the set of pixels corresponding to one or more objects that moved locally within the scene, the second set of positions being different than the first set of positions;
for each pixel in the set of pixels, compute a corresponding optical flow vector such that a plurality of optical flow vectors is computed, wherein the plurality of optical flow vectors map movement of the one or more objects, where the movement is represented by the set of pixels being at the first set of positions in the motion compensated history frame and being at the second set of positions in the current frame;
generate an adjusted history frame from the motion compensated history frame by applying the plurality of optical flow vectors to the set of pixels included in the motion compensated history frame to shift the set of pixels from being at the first set of positions in the motion compensated history frame to being at new positions such that the adjusted history frame reflects the new positions of the set of pixels, said new positions in the adjusted history frame corresponding to the second set of positions that were identified in the current frame for the set of pixels; and
after shifting the set of pixels to the new positions in the adjusted history frame, temporally filter the current frame with the adjusted history frame to generate a temporally filtered image.

### Claim 12 (depends on 11)

The computer system of claim 11, wherein, for content that was previously occluded in the motion compensated history frame by the set of pixels but that is now not occluded by the set of pixels in the adjusted history frame, image data for said content is obtained from the current frame, and
wherein, prior to obtaining said content from the current frame, the current frame is down-sampled to a lower resolution.

### Claim 13 (depends on 11)

The computer system of claim 11, wherein generating the temporally filtered image is performed by using weighted averages of the current frame and the adjusted history frame.

### Claim 14 (depends on 11)

The computer system of claim 11, wherein each optical flow vector in the plurality of optical flow vectors is a two-dimensional (2D) vector.

### Claim 15 (depends on 11)

The computer system of claim 11, wherein a first optical flow vector included in the plurality of optical flow vectors is different than a second optical flow vector included in the plurality of optical flow vectors.

### Claim 16 (depends on 11)

The computer system of claim 11, wherein shifting the set of pixels from being at the first set of positions in the motion compensated history frame to being at the new positions in the adjusted history frame is performed in two-dimensional (2D) space such that said shifting is a 2D transformation on a per pixel basis.

### Claim 17 (depends on 11)

The computer system of claim 11, wherein generating the current frame is performed at a rate of about 90 frames per second.

### Claim 18 (depends on 11)

The computer system of claim 11, wherein comparing the motion compensated history frame against the current frame to identify the set of pixels is performed using a block-matching algorithm.

### Claim 19

A computer system configured to generate a temporally filtered image that is designed to compensate for global motions of a camera and to compensate for local motions of one or more objects that are locally moving within a scene, said computer system comprising:
one or more processors; and
one or more computer-readable hardware storage devices that store instructions that are executable by the one or more processors to cause the computer system to:
acquire a history frame and a current frame of a scene, wherein the current frame is generated by a camera;
acquire inertial measurement unit (IMU) data that reflects a movement of the camera between a time when the history frame is acquired and a time when the current frame is generated by the camera;
in response using the IMU data to determine a first pose of the camera when the current frame was generated, perform a global motion compensation operation on the history frame to reproject a second pose of the history frame to match the first pose;
after performing the global motion compensation operation, compare the motion compensated history frame against the current frame to identify a set of pixels that are located at a first set of positions within the motion compensated history frame but that are located at a second set of positions within the current frame, the set of pixels corresponding to one or more objects that moved locally within the scene, the second set of positions being different than the first set of positions;
for each pixel in the set of pixels, compute a corresponding optical flow vector such that a plurality of optical flow vectors is computed, wherein the plurality of optical flow vectors map movement of the one or more objects, where the movement is represented by the set of pixels being at the first set of positions in the motion compensated history frame and being at the second set of positions in the current frame;
generate an adjusted history frame from the motion compensated history frame by applying the plurality of optical flow vectors to the set of pixels included in the motion compensated history frame to shift the set of pixels from being at the first set of positions in the motion compensated history frame to being at new positions such that the adjusted history frame reflects the new positions of the set of pixels, said new positions in the adjusted history frame corresponding to the second set of positions that were identified in the current frame for the set of pixels; and
after shifting the set of pixels to the new positions in the adjusted history frame, temporally filter the current frame with the adjusted history frame to generate a temporally filtered image.

### Claim 20 (depends on 19)

The computer system of claim 19, wherein a first optical flow vector included in the plurality of optical flow vectors is different than a second optical flow vector included in the plurality of optical flow vectors.
