# US 2016/0366309 A1 · Temporal and spatial video noise reduction

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0366309 A1
- Document ID: US20160366309A1
- Status: Published application
- Kind: A1
- Application number: 15/081,688
- Filed: 2016-03-25
- Published: 2016-12-15
- Earliest claimed: 2015-06-12
- Assignees: GoPro, Inc.
- Applicants: GoPro, Inc.
- Inventors: Bruno Cesar Douady-Pleven; Michael Serge André Kraak; Guillaume Matthieu Guerin; Thomas Nicolas Emmanuel Veit
- Web page: https://patentgrep.com/patent/US20160366309A1

## CPC

- G06T 3/4015: Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
- G06T 5/40: using histogram techniques
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/70: Denoising; Smoothing
- G06T 5/75: Unsharp masking
- G06T 5/92: based on global image properties
- G06T 5/94: based on local image properties, e.g. for local contrast enhancement
- H04N 5/213: Circuitry for suppressing or minimising impulsive noise (for suppressing or minimising disturbance in image signal generation H04N23/81)
- H04N 9/646: {for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters}
- H04N 9/77: Circuits for processing the brightness signal and the chrominance signal relative to each other, e.g. adjusting the phase of the brightness signal relative to the colour signal, correcting differential gain or differential phase (circuits for matrixing H04N9/67)
- H04N 23/741: by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
- H04N 23/81: for suppressing or minimising disturbance in the image signal generation
- H04N 25/618: for random or high-frequency noise
- H04N 25/626: Reduction of noise due to residual charges remaining after image readout, e.g. to remove ghost images or afterimages
- H04N 25/633: by using optical black pixels
- H04N 25/672: between adjacent sensors or output registers for reading a single image
- H04N 25/674: based on the scene itself, e.g. defocusing
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/20182: Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
- H04N 23/88: for colour balance, e.g. white-balance circuits or colour temperature control
- H04N 2209/046: Colour interpolation to calculate the missing colour values
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A system access a reference frame and temporally adjacent frames. For each portion of the reference image frame, the system calculates a pixel distance value between the portion of the reference image frame and a corresponding portion of each temporally adjacent image frame. If the pixel distance value indicates a potential ghosting artifact, the system computes a set of spatial noise reduction values for the image portion. Otherwise, the system computes a set of temporal noise reduction values for the image portion. The system blends the sets of computed spatial noise reduction values and the sets of computed temporal noise reduction values, and generates a modified reference image frame based on the blended set of noise reduction values.

## Claims

### Claim 1

A method for generating a noise reduced image captured by a camera comprising:
accessing a reference image frame from an ordered set of frames;
accessing image frames that are temporally adjacent to the reference image frame within the ordered set of frames;
for each portion of the reference image frame:
calculating a pixel distance value between the portion of the reference image frame and a corresponding portion of each temporally adjacent image frame;
in response to the pixel distance value indicating a potential ghosting artifact for the image portion, computing a set of spatial noise reduction values for the image portion based on image portions within the reference image frame adjacent to the image portion;
in response to the pixel distance value not indicating a potential ghosting artifact for the image portion, computing a set of temporal noise reduction values for the image portion based on the corresponding portions of each temporally adjacent image frame; and
blending the sets of computed spatial noise reduction values and the sets of computed temporal noise reduction values corresponding to the portions of the reference image frame based on the respective pixel value distances for the image portions to produce a set of blended noise reduction values; and
generating a modified reference image frame based on the blended set of noise reduction values.

### Claim 2 (depends on 1)

The method of claim 1, wherein calculating a pixel distance value between the portion of the reference image frame and a corresponding portion of each temporally adjacent image frame further comprises:
in response to determining that the signal to noise ratio (SNR) of the reference and temporally adjacent image frames exceeds a SNR threshold value:
identifying a first pixel value in the portion of the reference image frame;
identifying a second pixel value in the corresponding portion of one of the temporally adjacent image frames; and
determining the pixel value distance based on the difference between the first pixel value and the second pixel value; and
in response to determining that the signal to noise ratio (SNR) of the reference and temporally adjacent image frames does not exceed a SNR threshold value:
identifying a first blurred pixel value in the portion of the reference image frame, the blurred pixel value computed based on an average of the first pixel value and a first plurality of surrounding pixel values;
identifying a first blurred pixel value in the corresponding portion of one of the temporally adjacent image frames, the blurred pixel value computed based on an average of the second pixel value and a second plurality of surrounding pixel values; and
determining the pixel value distance based on the difference between the first blurred pixel value and the second blurred pixel value.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
determining that the pixel value difference based on the difference between the first blurred pixel value and the second blurred pixel value is beyond a threshold difference to the pixel value difference based on the difference between the first pixel value and the second pixel value;
determining that an aliasing error has occurred;
determining that a potential ghosting artifact exists for the image portion.

### Claim 4 (depends on 1)

The method of claim 1, wherein computing a set of spatial noise reduction values for the image portion further comprises:
determining as one of the set of spatial noise reduction values an average of a pixel value of a first pixel in the image portion in the reference frame with a plurality of pixel values of pixels surrounding the first pixel.

### Claim 5 (depends on 1)

The method of claim 1, wherein computing a set of spatial noise reduction values for the image portion further comprises:
determining as one of the set of spatial noise reduction values based on non-local means.

### Claim 6 (depends on 1)

The method of claim 1, wherein computing a set of temporal noise reduction values for the image portion further comprises:
determining as one of the set of temporal noise reduction values an average of a pixel value of a first pixel in the image portion in the reference frame with a plurality of pixel values of pixels at a corresponding position to the first pixel in the temporally adjacent image frames.

### Claim 7 (depends on 1)

The method of claim 1, wherein computing a set of spatial noise reduction values for the image portion further comprises:
determining a noise reduction strength of the set of temporal noise reduction values based on the number of frames averaged to generate the set of temporal noise reduction values; and
adjusting a set of coefficients for computing the set of spatial noise reduction values such that a noise reduction strength of the spatial noise reduction values is within a threshold range of the noise reduction strength of the set of temporal noise reduction values.

### Claim 8 (depends on 1)

The method of claim 1, wherein the blending the sets of computed spatial noise reduction values and the sets of computed temporal noise reduction values further comprises:
computing the set of blended noise reduction values based on the pixel value distance, such that the blended noise reduction values each image portion is a combination of the corresponding spatial noise reduction values and temporal noise reduction values weighted by the pixel value distance at that image portion, with higher pixel value distances giving relatively more weight to the spatial noise reduction values, and lower pixel value distances giving relatively more weight to the temporal noise reduction values.

### Claim 9 (depends on 1)

The method of claim 1, wherein the ordered set of frames are frames in a video captured by the camera.

### Claim 10

A specially-configured hardware system for generating a noise reduced image configured to:
access a reference image frame from an ordered set of frames;
access image frames that are temporally adjacent to the reference image frame within the ordered set of frames;
for each portion of the reference image frame:
calculate a pixel distance value between the portion of the reference image frame and a corresponding portion of each temporally adjacent image frame;
in response to the pixel distance value indicating a potential ghosting artifact for the image portion, compute a set of spatial noise reduction values for the image portion based on image portions within the reference image frame adjacent to the image portion;
in response to the pixel distance value not indicating a potential ghosting artifact for the image portion, compute a set of temporal noise reduction values for the image portion based on the corresponding portions of each temporally adjacent image frame; and
blend the sets of computed spatial noise reduction values and the sets of computed temporal noise reduction values corresponding to the portions of the reference image frame based on the respective pixel value distances for the image portions to produce a set of blended noise reduction values; and
generate a modified reference image frame based on the blended set of noise reduction values.

### Claim 11 (depends on 10)

The specially-configured hardware system of claim 10, further configured to:
in response to a determination that the signal to noise ratio (SNR) of the reference and temporally adjacent image frames exceeds a SNR threshold value:
identify a first pixel value in the portion of the reference image frame;
identify a second pixel value in the corresponding portion of one of the temporally adjacent image frames; and
determine the pixel value distance based on the difference between the first pixel value and the second pixel value; and
in response to a determination that the signal to noise ratio (SNR) of the reference and temporally adjacent image frames does not exceed a SNR threshold value:
identify a first blurred pixel value in the portion of the reference image frame, the blurred pixel value computed based on an average of the first pixel value and a first plurality of surrounding pixel values;
identify a first blurred pixel value in the corresponding portion of one of the temporally adjacent image frames, the blurred pixel value computed based on an average of the second pixel value and a second plurality of surrounding pixel values; and
determine the pixel value distance based on the difference between the first blurred pixel value and the second blurred pixel value.

### Claim 12 (depends on 11)

The specially-configured hardware system of claim 11, further configured to:
determine that the pixel value difference based on the difference between the first blurred pixel value and the second blurred pixel value is beyond a threshold difference to the pixel value difference based on the difference between the first pixel value and the second pixel value;
determine that an aliasing error has occurred;
determine that a potential ghosting artifact exists for the image portion.

### Claim 13 (depends on 10)

The specially-configured hardware system of claim 10, further configured to:
determine as one of the set of spatial noise reduction values an average of a pixel value of a first pixel in the image portion in the reference frame with a plurality of pixel values of pixels surrounding the first pixel.

### Claim 14 (depends on 10)

The specially-configured hardware system of claim 10, further configured to:
determine as one of the set of spatial noise reduction values based on non-local means.

### Claim 15 (depends on 10)

The specially-configured hardware system of claim 10, wherein further configured to:
determine as one of the set of temporal noise reduction values an average of a pixel value of a first pixel in the image portion in the reference frame with a plurality of pixel values of pixels at a corresponding position to the first pixel in the temporally adjacent image frames.

### Claim 16 (depends on 10)

The specially-configured hardware system of claim 10, further configured to:
determine a noise reduction strength of the set of temporal noise reduction values based on the number of frames averaged to generate the set of temporal noise reduction values; and
adjust a set of coefficients for computing the set of spatial noise reduction values such that a noise reduction strength of the spatial noise reduction values is within a threshold range of the noise reduction strength of the set of temporal noise reduction values.

### Claim 17 (depends on 10)

The specially-configured hardware system of claim 10, further configured to:
compute the set of blended noise reduction values based on the pixel value distance, such that the blended noise reduction values each image portion is a combination of the corresponding spatial noise reduction values and temporal noise reduction values weighted by the pixel value distance at that image portion, with higher pixel value distances giving relatively more weight to the spatial noise reduction values, and lower pixel value distances giving relatively more weight to the temporal noise reduction values.

### Claim 18 (depends on 10)

The specially-configured hardware system of claim 10, wherein the ordered set of frames are frames in a video captured by a camera.