# US 2010/0165152 A1 · Processing Images Having Different Focus > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2010/0165152 A1 - Document ID: US20100165152A1 - Status: Published application - Kind: A1 - Application number: 12/346,247 - Filed: 2008-12-30 - Published: 2010-07-01 - Earliest claimed: 2008-12-30 - Assignees: Massachusetts Institute of Technology - Inventors: Jae S. Lim - Web page: https://patentgrep.com/patent/US20100165152A1 ## CPC - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 11/60: Creating or editing images; Combining images with text - H04N 5/2621: {Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability} - H04N 23/743: Bracketing, i.e. taking a series of images with varying exposure conditions - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Processing images of a scene taken with different focus includes, for each of at least some of the images, determining at least one portion of the image having a predetermined characteristic. A representation of the scene is generated that includes an array of elements, where each of at least some of the elements corresponds to a determined portion of at least one of the images. An element of the array is based on a correspondence between a distance associated with a determined portion of at least one image and a focused distance associated with at least one image. The generated representation is stored or provided as output. ## Claims ### Claim 1 A method for processing images of a scene taken with different focus, comprising: for each of at least some of the images, determining at least one portion of the image having a predetermined characteristic; generating a representation of the scene comprising an array of elements, where each of at least some of the elements corresponds to a determined portion of at least one of the images, wherein an element of the array is based on a correspondence between a distance associated with a determined portion of at least one image and a focused distance associated with at least one image; and storing the generated representation or providing the generated representation as output. ### Claim 2 (depends on 1) The method of claim 1, wherein at least some of the elements of the array each represent an intensity for a given part of the scene. ### Claim 3 (depends on 1) The method of claim 1, wherein the generated representation of the scene comprises a composite image in which a portion of the composite image representing a given part of the scene is derived from at least one of the images in which the given part of the scene is most in focus. ### Claim 4 (depends on 1) The method of claim 1, wherein an element of the array is derived from a first portion of a first of the images that represents a given part of the scene, and the correspondence between a distance associated with the first portion of the first image and a focused distance associated with the first image indicates that the given part of the scene is more in focus in the first image than in any of the other images. ### Claim 5 (depends on 4) The method of claim 4, wherein the element of the array comprises the first portion of the first image. ### Claim 6 (depends on 4) The method of claim 4, wherein the element of the array is derived from an interpolation between the first portion of the first image and a corresponding second portion of a second of the images. ### Claim 7 (depends on 1) The method of claim 1, wherein at least some of the elements of the array each represent a distance to a given part of the scene. ### Claim 8 (depends on 1) The method of claim 1, wherein the generated representation of the scene comprises a depth map in which each distance associated with a given part of the scene is derived from a focused distance associated with at least one of the images in which the given part of the scene is most in focus. ### Claim 9 (depends on 8) The method of claim 8, wherein generating the depth map comprises adjusting an initial distance estimate associated with the given part of the scene based on the focused distance. ### Claim 10 (depends on 9) The method of claim 9, wherein the initial distance estimate associated with the given part of the scene is received from an initial depth map of the scene. ### Claim 11 (depends on 1) The method of claim 1, wherein determining at least one portion of a given image having a predetermined characteristic comprises measuring a local characteristic of each of multiple portions of the given image. ### Claim 12 (depends on 11) The method of claim 11, wherein the local characteristic comprises local variation in intensity. ### Claim 13 (depends on 12) The method of claim 12, wherein an element of the array representing a distance corresponds to portions of the images representing a given part of the scene, and the element of the array is derived from a focused distance associated with a given image for which the local variation of the corresponding portion is larger than the local variation of corresponding portions of the other images. ### Claim 14 (depends on 12) The method of claim 12, wherein measuring the local characteristic comprises calculating a value of a two-dimensional variance function. ### Claim 15 (depends on 1) The method of claim 1, wherein determining at least one portion of a given image having a predetermined characteristic comprises comparing a distance associated with the given portion of the given image to a focused distance associated with the given image. ### Claim 16 (depends on 15) The method of claim 15, wherein the distance associated with the given portion of the given image is derived from a depth map of the scene. ### Claim 17 (depends on 15) The method of claim 15, wherein an element of the array is derived from a portion of an image for which the focused distance is closer to the associated distance than the focused distances associated with the other images. ### Claim 18 (depends on 1) The method of claim 1, wherein each of at least some of the portions of an image comprises one or more pixels of a digital image. ### Claim 19 (depends on 1) The method of claim 1, further comprising capturing each of the images of the scene with a different focus. ### Claim 20 (depends on 19) The method of claim 19, further comprising storing information indicating the focus at which each of the images was captured. ### Claim 21 (depends on 1) The method of claim 1, wherein each of the images represents substantially the same view of the scene. ### Claim 22 A method for generating an image of a scene, comprising: receiving a first representation of the scene comprising a first array of elements that each represents an intensity for a given part of the scene; receiving a second representation of the scene comprising a second array of elements that each represents a distance to a given part of the scene; generating an image of the scene in which each portion of the generated image is based on an element from the first array and an element from the second array that represent substantially the same part of the scene; and storing the generated image or providing the generated image as output. ### Claim 23 (depends on 22) The method of claim 22, wherein generating the image of the scene comprises applying a blurring function to a plurality of intensity values from the first array based on at least one distance value from the second array and a target focused distance. ### Claim 24 (depends on 23) The method of claim 23, wherein generating the image of the scene comprises applying a blurring function to a plurality of intensity values from the first array based on a difference between a distance value from the second array and a target focused distance. ### Claim 25 (depends on 24) The method of claim 24, wherein the distance value from the second array represents a distance to a part of the scene represented by at least one of the plurality of intensity values to which the blurring function is being applied. ### Claim 26 A computer-readable medium storing a computer program for processing images of a scene taken with different focus, the computer program comprising instructions for causing a computer system to: for each of at least some of the images, determine at least one portion of the image having a predetermined characteristic; generate a representation of the scene comprising an array of elements, where each of at least some of the elements corresponds to a determined portion of at least one of the images, wherein an element of the array is based on a correspondence between a distance associated with a determined portion of at least one image and a focused distance associated with at least one image; and store the generated representation or provide the generated representation as output. ### Claim 27 A system, comprising: an image capture subsystem configured to capture images of a scene taken with different focus; and an image processing subsystem configured to, for each of at least some of the images, determine at least one portion of the image having a predetermined characteristic, and generate a representation of the scene comprising an array of elements, where each of at least some of the elements corresponds to a determined portion of at least one of the images, wherein an element of the array is based on a correspondence between a distance associated with a determined portion of at least one image and a focused distance associated with at least one image.