# US 8,570,386 B2 · Method of merging images and relative method of generating an output image of enhanced quality > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,570,386 B2 - Document ID: US8570386B2 - Status: Granted - Kind: B2 - Application number: 12/319,001 - Filed: 2008-12-31 - Published: 2010-07-01 - Granted: 2013-10-29 - Earliest claimed: 2008-12-31 - Assignees: STMicroelectronics S.r.l. - Applicants: Alfio Castorina; Giuseppe Spampinato; Alessandro Capra; Arcangelo Ranieri Bruna - Inventors: Alfio Castorina; Giuseppe Spampinato; Alessandro Capra; Arcangelo Ranieri Bruna - Web page: https://patentgrep.com/patent/US8570386B2 ## CPC - H04N 23/68: for stable pick-up of the scene, e.g. compensating for camera body vibrations - G06T 5/20: using local operators - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 5/70: Denoising; Smoothing - G06T 5/73: Deblurring; Sharpening - H04N 23/84: for processing colour signals - H04N 25/134: {based on three different wavelength filter elements} - G06T 2207/10024: Color image - G06T 2207/20182: Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering - G06T 2207/20201: Motion blur correction - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method of processing and device configure to process digital images to enhance image quality and correct motion blur. A number N of images of a scene are captured with an exposure time T. An order of sharpness of the images is determined and the sharpest image is used as a reference image for generating an output image. ## Claims ### Claim 1 A method, comprising: merging a number of two or more images N of a same scene into an output image in an image processing system, by, determining an order of sharpness among the N images, the determining including identifying a sharpest image in the N images; determining vertical and horizontal offsets with respect to the sharpest image in the N images for other images in the N images and generating corresponding offset-free images; generating pixels of an intermediate image by combining corresponding pixels of the sharpest image and the offset-free images, the generating pixels of the intermediate image including calculating a weight of a pixel of an offset-free image based on a calculated standard deviation of noise and an absolute value of a difference between an intensity of the pixel and of a respective pixel of the sharpest image; and filtering noise from the intermediate image to generate the output image, the determining, generating and filtering being performed by one or more configured devices of the image processing system. ### Claim 2 (depends on 1) The method of claim 1 wherein generating pixels of the intermediate image comprises computing a weighted average by assigning larger weights for sharper images and smaller weights for less sharp images. ### Claim 3 (depends on 1) The method of claim 1 wherein generating pixels of the intermediate image comprises: calculating the standard deviation of noise by processing pixels of a set of the N images; for each pixel of the offset-free images, calculating a respective weight in function of the standard deviation and in function of an absolute value of a difference between an intensity of the pixel and of a respective pixel of the sharpest image; and generating each pixel of the intermediate image as a weighted average with the weights of the corresponding pixels of the offset-free images. ### Claim 4 (depends on 3) The method of claim 3 wherein the set of the N images comprises each of the N images. ### Claim 5 (depends on 1) The method of claim 1 wherein generating pixels of the intermediate image comprises: performing temporal noise reduction iteratively applied in order starting from a sharpest offset-free image to a least sharp offset-free image. ### Claim 6 (depends on 1) The method of claim 1 wherein the images are Bayer images and the offsets are calculated considering only green pixels. ### Claim 7 A method, comprising: generating an output image having a level of illumination corresponding to an exposure time T, by, taking a plurality of N images of a same scene over an exposure time T/N; amplifying a luminance of each of the images up to the level of illumination; determining an order of sharpness among the N images, the determining including identifying a sharpest image in the N images; determining vertical and horizontal offsets with respect to the sharpest image in the N images for other images in the N images and generating corresponding offset-free images; generating pixels of an intermediate image by combining corresponding pixels of the sharpest image and the offset-free images, the generating pixels of the intermediate image including calculating a weight of a pixel of an offset-free image based on a calculated standard deviation of noise and an absolute value of a difference between an intensity of the pixel and of a respective pixel of the sharpest image; and filtering noise from the intermediate image to generate the output image, the determining, the generating and the filtering being performed by a configured image processing system. ### Claim 8 (depends on 7) The method of claim 7 wherein generating pixels of the intermediate image comprises computing a weighted average by assigning larger weights for sharper images and smaller weights for less sharp images. ### Claim 9 (depends on 7) The method of claim 7 wherein generating pixels of the intermediate image comprises: calculating the standard deviation of noise by processing pixels of a set of the plurality of images; for each pixel of the offset-free images, calculating a respective weight in function of the standard deviation and in function of an absolute value of a difference between an intensity of the pixel and of a respective pixel of the sharpest image; and generating each pixel of the intermediate image as a weighted average with the weights of the corresponding pixels of the offset-free images. ### Claim 10 (depends on 7) The method of claim 7 wherein generating pixels of the intermediate image comprises: performing temporal noise reduction iteratively applied in order starting from a sharpest offset-free image to a least sharp offset-free image. ### Claim 11 (depends on 7) The method of claim 7 wherein the images are Bayer images and the offsets are calculated considering only green pixels. ### Claim 12 A device to process a set of two or more images of a scene, comprising: a sensor configured to generate the set of images of the scene; a memory configured to store the set of images and respective characteristic curves; a raster processing block configured to generate respective signals representing a noise rms value and a sharpness parameter for each image in the set; an ordering block configured to assign an order to each image in the set based on the sharpness parameters, the ordering indicating a sharpest image in the set of images; a stabilization block configured to generate signals representing horizontal and vertical offsets for images with respect to the sharpest image in the set based on characteristic curves; and an image processing pipeline having a noise reduction block configured to generate an output image filtered from noise, and an image generation block configured to generate an RGB image corresponding to the set of images, wherein the image processing pipeline is configured to assign a pixel weight based on a noise rms value and on an absolute value of a difference between an intensity of the pixel and of a respective pixel of the sharpest image. ### Claim 13 (depends on 12) The device of claim 12 wherein the images are Bayer images. ### Claim 14 A system to process a set of two or more digital images, comprising: a sharpness block configured to determine a sharpest image in the set of images; an image stabilization block configured to compensate for motion in the set of images using the sharpest image in the set of images as a reference by determining vertical and horizontal offsets with respect to the sharpest image for other images in the set of images and generating corresponding offset-free images; and an image processing block configured to: generate pixels of an intermediate image by combining corresponding pixels of the sharpest image and the offset-free images, the generating pixels of the intermediate image including calculating a weight of a pixel of an offset-free image based on a calculated standard deviation of noise and on an absolute value of a difference between an intensity of the pixel and of a respective pixel of the sharpest image; and reduce noise in an output image generated from the set of images. ### Claim 15 (depends on 14) The system of claim 14, further comprising an image sensor configured to generate the set of images. ### Claim 16 (depends on 14) The system of claim 14 wherein the sharpness block is configured to order the set of images in decreasing order of sharpness. ### Claim 17 (depends on 14) The system of claim 14, further comprising a curve generator configured to generate characteristic curves for images in the set of images, wherein the stabilization block is configured to compensate for the motion based on the characteristic curves. ### Claim 18 (depends on 14) The system of claim 14, wherein the images are Bayer images. ### Claim 19 A non-transitory computer readable memory medium comprising contents that cause a computing device to implement a method of merging a plurality of N images into a single output image, the method including: determining an order of sharpness among the N images, the determining identifying a sharpest image among the N images; determining vertical and horizontal offsets with respect to the sharpest image in the N images for other images in the N images and generating corresponding offset-free images; generating pixels of an intermediate image by combining corresponding pixels of the sharpest image and the offset-free images, the generating pixels of the intermediate image including calculating a weight of a pixel of an offset-free image based on a calculated standard deviation of noise and on an absolute value of a difference between an intensity of the pixel and of a respective pixel of the sharpest image; and filtering noise from the intermediate image to generate the output image. ### Claim 20 (depends on 19) The non-transitory computer readable memory medium of claim 19, wherein the method further comprises: generating the N images by taking a plurality of N images of a same scene; and amplifying a luminance of each of the images N up to a level of illumination.