# US 2021/0241427 A1 · Temporal de-noising > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0241427 A1 - Document ID: US20210241427A1 - Status: Published application - Kind: A1 - Application number: 16/768,870 - Filed: 2018-12-02 - Published: 2021-08-05 - Earliest claimed: 2015-03-21 - Assignees: Mine One GmbH - Applicants: MINE ONE GmbH - Inventors: JAMES A. MCCOMBE; CHRISTOPH BIRKHOLD - Web page: https://patentgrep.com/patent/US20210241427A1 ## CPC - G06T 5/70: Denoising; Smoothing - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 7/593: from stereo images - H04N 23/683: {performed by a processor, e.g. controlling the readout of an image memory} - H04N 23/84: for processing colour signals - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - G06T 2207/10012: Stereo images - G06T 2207/10024: Color image - G06T 2207/20182: Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering - G06T 2207/20221: Image fusion; Image merging - H04N 9/64: Circuits for processing colour signals (H04N9/77 takes precedence; camera processing pipelines for processing colour signals H04N23/84) - H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Methods, systems, devices and computer software/program code products enable generating reduced-noise image frames based on image frames received from a digital, camera pipeline; and enable efficient stereo image search between corresponding images generated by at least two cameras. ## Claims ### Claim 1 A method of generating reduced-noise image frames based on image frames received from a digital camera pipeline, the method comprising: in a digital processing resource comprising at least one digital processor: A. receiving at least one frame of image pixel data from a digital camera pipeline; B. receiving image metadata corresponding to the at least one received frame; and C. executing temporal de-noising operations for at least selected pixels of the at least one received frame of image pixel data, wherein executing temporal dc-noising operations comprises: (1) evaluating a noise model to estimate the noise present in a selected pixel from the at least one received frame; (2) blending data from a persistent image buffer with data from the at least one received frame; (3) updating data in the persistent image buffer; and (4) outputting the blended pixel data to a reduced-noise frame buffer, thereby to generate a reduced-noise image frame. ### Claim 2 (depends on 1) The method of claim 1 wherein the persistent image buffer aligns with a camera frame buffer, such that there exists a substantially one-to-one correspondence between a given frame buffer pixel and an element in the persistent image buffer, when each is addressed in the same coordinate space. ### Claim 3 (depends on 1) The method of claim 1 wherein the persistent image buffer aligns with a post-transformed image buffer, such that a selected transform relates a given frame buffer pixel and an element in the persistent image buffer. ### Claim 4 (depends on 3) The method of claim 3 wherein the transform is a rectification operation. ### Claim 5 (depends on 3) The method of claim 3 wherein the transform comprises any of pixel stretch or blending operations, re-projection, correction for lens effects, or transformation enabling the persistent image buffer to represent data in a format different from the data format of the frame buffer. ### Claim 6 (depends on 3) The method of claim 3 wherein the transform can be changed on a frame-by-frame basis. ### Claim 7 (depends on 1) The method of claim 1 wherein the persistent image buffer is an abstract data structure, and wherein correspondence between a given frame buffer pixel and an element in the persistent image buffer is of arbitrary complexity. ### Claim 8 (depends on 7) The method of claim 7 wherein correspondence between a given frame buffer pixel and an element in the persistent image buffer comprises a lookup operation into a data structure. ### Claim 9 (depends on 8) The method of claim 8 wherein the lookup operation comprises at least one level of indirection. ### Claim 10 (depends on 1) The method of claim 1 wherein the noise model comprises a simulation of the inverse behavior of at least selected aspects of the digital camera pipeline. ### Claim 11 (depends on 10) The method of claim 10 wherein the noise model comprises a simulation of the inverse of the behavior of a variable gain amplifier. ### Claim 12 (depends on 10) The method of claim 10 wherein the noise model comprises a simulation of the inverse of the behavior of automatic white-balance logic. ### Claim 13 (depends on 10) The method of claim 10 wherein the noise model comprises a simulation of the inverse of the behavior of automatic exposure control. ### Claim 14 (depends on 10) The method of claim 10 wherein the noise model comprises a simulation of the number of electrons in an electron well associated with a light sensing element of the digital camera pipeline. ### Claim 15 (depends on 1, 10) The method of claim 1 or 10 in which evaluating a noise model comprises interpolating between coefficients in a lookup table. ### Claim 16 (depends on 15) The method of claim 15 in which at least one lookup table is generated in response to metadata associated with at least one received frame. ### Claim 17 (depends on 1, 10) The method of claim 1 or 10 wherein the noise model is evaluated in a manner dependent on the color represented by image pixel data. ### Claim 18 (depends on 1) The method of claim 1 in which the persistent image buffer comprises a summation of data from prior frames. ### Claim 19 (depends on 18) The method of claim 18 in which the persistent image buffer comprises an average of data from prior frames. ### Claim 20 (depends on 1, 10, 18, 19) The method of claim 1, 10, 18 or 19 in which at least one pixel in the persistent image buffer is invalidated in response to a selected result of the evaluation of the noise model. ### Claim 21 (depends on 1, 10, 18, 19) The method of claim 1, 10, 18 or 19 in which at least one pixel in the persistent image buffer is invalidated in response to a selected result of a comparison with a pixel from the at least one received frame. ### Claim 22 (depends on 1, 10) The method of claim 1 or 10 in which the persistent image buffer is re-projected in response to camera movement. ### Claim 23 (depends on 22) The method of claim 22 in which the re-projection is a selected image warp in response to camera rotation. ### Claim 24 (depends on 22) The method of claim 22 in which the re-projection utilizes a depth buffer. ### Claim 25 (depends on 1, 10, 18, 19) A system for executing stereo image search between corresponding images generated by at least two cameras, the system comprising: A. at least two temporal do-noising pipelines operable according to the method of any of claims 1, 10, 18 or 19 each de-noising pipeline being operable to receive image frames from a respective camera; B. at least one digital memory element, the memory element being operable to receive output images from the respective de-noising pipelines; and C. a stereo correspondence search module, the stereo correspondence search module being operable to compare an output image from a first temporal de-noising pipeline with a tentatively corresponding output image from another temporal de-noising pipeline. ### Claim 26 (depends on 25) The system of claim 25 wherein the temporal dc-noising pipelines comprise processing modules; the processing modules comprising software modules executing on computational resources. ### Claim 27 (depends on 26) The system of claim 26 wherein the software modules execute on shared computational resources. ### Claim 28 (depends on 25) The system of claim 25 wherein the temporal de-noising pipelines comprise dedicated electronic components. ### Claim 29 (depends on 25) The system of claim 25 wherein at least two cameras participate in a shutter synchronization protocol. ### Claim 30 A system for generating reduced-noise image frames based on image frames received from a digital camera pipeline, the system comprising: a digital processing resource comprising at least one digital processor, the digital processing resource being operable to: A. receive at least one frame of image pixel data from a digital camera pipeline; B. receive image metadata corresponding to the at least one received frame; and C. execute temporal de-noising operations for at least selected pixels of the at least one received frame of image pixel data, wherein executing temporal de-noising operations comprises: (1) evaluating a noise model to estimate the noise present in a selected pixel from the at least one received frame; (2) blending data from a persistent image buffer with data from the at least one received frame; (3) updating data in the persistent image buffer, and (4) outputting the blended pixel data to a reduced-noise frame buffer, thereby to generate a reduced-noise image frame. ### Claim 31 A program product for use with a digital processing system to enable the digital processing system to generate reduced-noise image frames based on image frames received from a digital camera pipeline, the digital processing system comprising a digital processing resource, the digital processing resource comprising at least one digital processor, the program product comprising digital processor-executable program instructions stored on a non-transitory digital processor-readable medium, which when executed in the digital processing resource cause the digital processing resource to: A. receive at least one frame of image pixel data from a digital camera pipeline; B. receive image metadata corresponding to the at least one received frame; and C. execute temporal de-noising operations for at least selected pixels of the at least one received frame of image pixel data, wherein executing temporal de-noising operations comprises: (1) evaluating a noise model to estimate the noise present in a selected pixel from the at least one received frame; (2) blending data from a persistent image buffer with data from the at least one received frame; (3) updating data in the persistent image buffer; and (4) outputting the blended pixel data to a reduced-noise frame buffer, thereby to generate a reduced-noise image frame. ### Claim 32 (depends on 1, 10, 18, 19) A method for executing stereo image search between corresponding images generated by at least two cameras, the method comprising: A. utilizing at least two de-noising pipelines, operable according to the method of any of claims 1, 10, 18 or 19 and each operable to receive image frames from a respective camera; B. utilizing at least one digital memory element to receive output images from the respective de-noising pipelines; and C. utilizing a stereo correspondence module to compare an output image from a first temporal de-noising pipeline with a tentatively corresponding output image from another temporal dc-noising pipeline. ### Claim 33 (depends on 1, 10, 18, 19) A program product for use with a digital processing system to enable the digital processing system to execute stereo image search between corresponding images generated by at least two cameras, the digital processing system comprising a digital processing resource, the digital processing resource comprising at least one digital processor, the program product comprising digital processor-executable program instructions stored on a non-transitory digital processor-readable medium, which when executed in the digital processing resource cause the digital processing resource to: A. utilize at least two temporal de-noising pipelines, each operable according to the method of any of claims 1, 10, 18 or 19 to receive image frames from a respective camera; B. utilize at least one digital memory element to receive output images from the respective de-noising pipelines; and C. utilize a stereo correspondence search module to compare an output image from a first temporal de-noising pipeline with a tentatively corresponding output image from another temporal de-noising pipeline. ### Claim 34 (depends on 1) The method of claim 1 further comprising: utilizing a persistent data store to retain information representing the number of valid samples. ### Claim 35 (depends on 34) The method of claim 34 further comprising: utilizing the persistent data store to retain information estimating the number of electrons or photons that have been integrated into corresponding data. ### Claim 36 (depends on 1) The method of claim 1 further comprising: utilizing information from the noise model to evaluate pixel data to quantify the likelihood that a measured variation in the pixel data is due to noise rather than a change in observed situation. ### Claim 37 (depends on 1) The method of claim 1 further comprising: applying an output from the noise model to a video signal, to improve suitability of the signal as an input to a stereo disparity search operation. ### Claim 38 (depends on 1) The method of claim 1 wherein buffer data comprises raw captured pixel information. ### Claim 39 (depends on 1) The method of claim 1 wherein image metadata comprises exposure information from an auto-exposure element. ### Claim 40 (depends on 1) The method of claim 1 wherein image metadata comprises white-balance information from an automatic white-balance element. ### Claim 41 (depends on 1) The method of claim 1 wherein image metadata comprises any of black level information, variable gain amplification information, or exposure time information. ### Claim 42 (depends on 1) The method of claim 1 wherein image metadata comprises information generated by a digital camera pipeline. ### Claim 43 (depends on 1) The method of claim 1 wherein image metadata comprises information describing observations made by sensors. ### Claim 44 (depends on 43) The method of claim 43 wherein the sensors comprise at least one camera. ### Claim 45 (depends on 1) The method of claim 1 wherein image metadata comprises information describing the magnitude or presence of actions executed by an imaging pipeline. ### Claim 46 (depends on 45) The method of claim 45 wherein an action executed by an imaging pipeline is a filter operation. ### Claim 47 (depends on 19) The method of claim 19 further comprising: utilizing a cumulative average, in which for a given pixel in a captured frame, a captured pixel value is added to a persistent pixel value, and an effective sample count for the persistent pixel is incremented, and wherein a pixel value for a de-noised frame is produced by dividing the sum of pixel values in the persistent pixel by the associated effective sample count for that pixel. ### Claim 48 (depends on 47) The method of claim 47 further comprising: providing persistent tracking of the statistical deviation for each pixel value from one captured frame to a next captured frame. ### Claim 49 (depends on 19) The method of claim 19 further comprising: utilizing a running average, wherein the standard deviation for a given pixel is utilized to determine a weight to assign to a newly captured pixel value, and a weight to assign to previously captured pixel values, to enable blending the captured pixel value and a persistent pixel value in accordance with the weights, to create a de-noised output pixel value. ### Claim 50 (depends on 49) The method of claim 49 wherein the noise model is a source for blending information useable in executing the blending. ### Claim 51 (depends on 1) The method of claim 1 wherein data corresponding to a captured pixel is blended with data from a corresponding persistent pixel in accordance with a blending ratio, and wherein the blending ratio is influenced by metadata from a corresponding captured frame. ### Claim 52 (depends on 51) The method of claim 51 wherein the metadata from a corresponding captured frame comprises amplification information. ### Claim 53 (depends on 51) The method of claim 51 further wherein data from a given pixel is normalized with respect to metadata values from a corresponding captured frame, prior to blending. ### Claim 54 (depends on 1) The method of claim 1 further comprising: executing a compression operation on data from a persistent image buffer. ### Claim 55 (depends on 54) The method of claim 54 wherein the compression is lossless. ### Claim 56 (depends on 54) The method of claim 54 wherein the compression is lossy. ### Claim 57 (depends on 54) The method of claim 54 wherein the compression operation is executed in accordance with a selected compression scheme. ### Claim 58 (depends on 57) The method of claim 57 wherein the selected compression scheme is based at least in part on an observation that spatially proximate pixels often have similar values. ### Claim 59 (depends on 22) The method of claim 22 wherein the re-projecting utilizes pixel depth values to increase accuracy of re-projection. ### Claim 60 (depends on 59) The method of claim 59 wherein a pixel depth value is estimated from stereo disparity information. ### Claim 61 (depends on 59) The method of claim 59 wherein a pixel depth value is estimated from information derived from any of a depth sensor, time-of-flight sensor, or structured-light based depth sensor. ### Claim 62 (depends on 22) The method of claim 22 wherein the re-projecting utilizes camera transformation information generated by additional sensors. ### Claim 63 (depends on 62) The method of claim 62 wherein the additional sensors comprise any of at least one accelerometer, gyroscope, or additional camera or set of cameras. ### Claim 64 (depends on 22) The method of claim 22 wherein the re-projecting utilizes camera transformation information generated by a process that interprets data from the same camera or cameras that capture image data, to generate camera transformation information. ### Claim 65 (depends on 64) The method of claim 64 wherein the process that interprets data comprises any of simultaneous location and mapping (SLAM), parallel tracking and mapping (PTAM), or dense tracking and mapping (DTAM). ### Claim 66 (depends on 22) The method of claim 22 wherein (a) the re-projecting utilizes camera transformation information resulting from a stereo disparity search executed on de-noised output data, (b) the camera transformation information results from evaluation across a captured frame, and (c) the camera transformation information is utilized to re-project the persistent image buffer stored prior to camera movement. ### Claim 67 (depends on 1) The method of claim 1 further comprising: constructing a lookup table that can be used for all pixel values in a given frame. ### Claim 68 (depends on 1) The method of claim 1 further comprising: executing the de-noising operations utilizing computational resources within the digital camera pipeline. ### Claim 69 (depends on 68) The method of claim 68 wherein the computational resources within the digital camera pipeline comprise dedicated hardware elements. ### Claim 70 (depends on 69) The method of claim 69 further comprising: utilizing analog information directly generated by the digital camera pipeline. ### Claim 71 (depends on 1) The method of claim 1 further comprising: for a given pixel, interpreting each color component value independently of other color component values. ### Claim 72 (depends on 71) The method of claim 71 further comprising: estimating noise in one color channel of a given pixel without regard to values of other color channels for the given pixel. ### Claim 73 (depends on 1) The method of claim 1 further comprising: constructing a statistical noise model having different parameters for different color channels, to account for differences in sensitivity between different color channels of a given digital camera. ### Claim 74 (depends on 73) The method of claim 73 wherein the different parameters for different color channels comprise a different electron well capacity for a given color channel. ### Claim 75 (depends on 73) The method of claim 73 wherein the different parameters for different color channels comprise: allocating additional sampling to a given color channel. ### Claim 76 (depends on 73) The method of claim 73 wherein the different parameters for different color channels comprise: executing a selected transformation or adjustment differently for a given color channel. ### Claim 77 (depends on 1) The method of claim 1 further comprising: for a given pixel, utilizing a combined pixel value encoding a compound color value. ### Claim 78 (depends on 77) The method of claim 77 further comprising: for the given pixel, utilizing an independent-color color space and reducing the pixel value to a scalar quantity. ### Claim 79 (depends on 78) The method of claim 78 wherein the independent-color color space is an RGB color space. ### Claim 80 (depends on 78) The method of claim 78 wherein reducing of the pixel value to a scalar quantity is executed by calculating the magnitude of the corresponding 3-dimensional color vector using the Pythagorean theorem. ### Claim 81 (depends on 78) The method of claim 78 wherein reducing of the pixel value to a scalar quantity is executed by utilizing only the value of the largest component of the corresponding color vector. ### Claim 82 (depends on 77) The method of claim 77 further comprising: approximating a color vector magnitude. ### Claim 83 (depends on 82) The method of claim 82 further comprising: utilizing a combined-color color space and executing noise estimation utilizing the luminance channel alone. ### Claim 84 (depends on 83) The method of claim 83 wherein the combined-color color space is a YUV color space.