# US 12,002,233 B2 · Systems and methods for estimating depth and visibility from a reference viewpoint for pixels in a set of images captured from different viewpoints > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,002,233 B2 - Document ID: US12002233B2 - Status: Granted - Kind: B2 - Application number: 17/162,542 - Filed: 2021-01-29 - Published: 2021-05-20 - Granted: 2024-06-04 - Earliest claimed: 2012-08-21 - Assignees: Adeia Imaging LLC - Applicants: Adeia Imaging LLC - Inventors: Florian Ciurea; Kartik Venkataraman; Gabriel Molina; Dan Lelescu - Web page: https://patentgrep.com/patent/US12002233B2 ## CPC - G06T 7/593: from stereo images - G02B 27/0075: {with means for altering, e.g. increasing, the depth of field or depth of focus} - G06T 7/557: from light fields, e.g. from plenoptic cameras - G06T 7/85: {Stereo camera calibration} - G06T 15/20: Perspective computation - H04N 13/128: Adjusting depth or disparity - H04N 13/232: using fly-eye lenses, e.g. arrangements of circular lenses - H04N 13/243: using three or more two-dimensional [2D] image sensors - H04N 23/16: Optical arrangements associated therewith, e.g. for beam-splitting or for colour correction - G06T 2200/21: involving computational photography - G06T 2207/10012: Stereo images - G06T 2207/10024: Color image - G06T 2207/10052: Images from lightfield camera - H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals} - H04N 2013/0088: {Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Systems in accordance with embodiments of the invention can perform parallax detection and correction in images captured using array cameras. Due to the different viewpoints of the cameras, parallax results in variations in the position of objects within the captured images of the scene. Methods in accordance with embodiments of the invention provide an accurate account of the pixel disparity due to parallax between the different cameras in the array, so that appropriate scene-dependent geometric shifts can be applied to the pixels of the captured images when performing super-resolution processing. In a number of embodiments, generating depth estimates considers the similarity of pixels in multiple spectral channels. In certain embodiments, generating depth estimates involves generating a confidence map indicating the reliability of depth estimates. ## Claims ### Claim 1 A method of estimating distances to objects within a scene from a light field comprising a set of images captured from different viewpoints using a processor configured by an image processing application, the method comprising: selecting a reference image from a reference viewpoint relative to the viewpoints of the set of images captured from different viewpoints; determining depth estimates for pixel locations in the reference image using at least a subset of the set of images, where a depth estimate for a given pixel location in the reference image is determined by: dividing the reference image into spatial blocks; generating depth measurements for a sparser subset of indicator pixels within the spatial blocks, where the spatial block includes a plurality of indicator pixels; assigning a depth to the pixels in each of the spatial blocks based upon the depths of the indicator pixels. ### Claim 2 (depends on 1) The method of claim 1, further comprising: determine a reliability of the depth assigned to each of the spatial blocks; when a depth transition is detected within spatial blocks, refining the depth of the spatial block by performing additional depth measurements within the spatial bock that includes the depth transitions. ### Claim 3 (depends on 1) The method of claim 1, wherein a depth of a non-indicator pixels is calculated as a normalized weighted average of the distances to the nearest indicator pixels within a fixed neighborhood. ### Claim 4 (depends on 1) The method of claim 1, further comprising using nearest neighbor interpolation to assign depths to the pixels in the spatial block based upon the depth measurements of the indicator pixels. ### Claim 5 (depends on 1) The method of claim 1, wherein assigning a depth to the pixels in each of the spatial blocks is obtained through interpolation of neighboring indicator pixels. ### Claim 6 (depends on 5) The method of claim 5, wherein weights for the interpolation incorporate intensity similarity and spatial distance to the nearest indicator pixel. ### Claim 7 (depends on 1) The method of claim 1, further comprising measuring similarity of corresponding pixels using a block-based similarity measure. ### Claim 8 (depends on 1) The method of claim 1, wherein selecting a reference viewpoint relative to the viewpoints of the set of images captured from different viewpoints comprises selecting a viewpoint from the set consisting of: the viewpoint of one of the images; and a virtual viewpoint. ### Claim 9 (depends on 1) The method of claim 1, further comprising: identifying pixels as corresponding pixels in the at least a subset of the set of images that correspond to the given pixel location in the reference image based upon expected disparity at a plurality of depths, wherein disparity determines an amount an object shifts between images from different cameras and is based on a baseline distance between the different cameras; comparing a similarity of intensity values of the corresponding pixels identified in the subset of the set of images at each of the plurality of depths; and selecting the depth from the plurality of depths at which the identified corresponding pixels have the highest degree of similarity as a depth estimate for the given pixel location in the reference image. ### Claim 10 (depends on 9) The method of claim 9, wherein comparing a similarity of intensity values of the corresponding pixels comprises using a cost function to determine the similarity of corresponding pixels. ### Claim 11 (depends on 9) The method of claim 9, wherein identifying corresponding pixels comprises determining that a pixel in a given image from the set of images corresponds to the pixel location in the reference image from the reference viewpoint by applying a scene dependent shift to the pixel location in the reference image from the reference viewpoint that is determined based upon: a depth estimate of the pixel location in the reference image; and a baseline between the viewpoint of the given image and the reference viewpoint. ### Claim 12 (depends on 9) The method of claim 9, further comprising generating confidence metrics for the current depth estimates for pixel locations in the reference image from the reference viewpoint, wherein a confidence metric for a particular pixel location is based on a signal to noise ratio, SNR, in a region of the particular pixel location with which a depth estimate is associated. ### Claim 13 (depends on 9) The method of claim 9, wherein selecting the depth from the plurality of depths at which the identified corresponding pixels have the highest degree of similarity as the depth estimate for the given pixel location in the reference image from the reference viewpoint further comprises selecting the depth from the plurality of depths at which the cost function for the identified corresponding pixels indicates the highest level of similarity. ### Claim 14 (depends on 9) The method of claim 9, further comprising fusing pixels from the set of images using the processor configured by the image processing application based upon the depth estimates to create a fused image having a resolution that is greater than the resolutions of the images in the set of images. ### Claim 15 (depends on 14) The method of claim 14, further comprising synthesizing an image from the reference viewpoint using the processor configured by the image processing application to perform a super-resolution process based upon the fused image from the reference viewpoint, the set of images captured from different viewpoints, and the depth estimates. ### Claim 16 (depends on 1) The method of claim 1, further comprising: normalizing the set of images, wherein the normalizing comprises using photometric calibration data to perform photometric normalization. ### Claim 17 (depends on 16) The method of claim 16, wherein normalizing the set of images further comprises: utilizing calibration information to correct for photometric variations and scene-independent geometric distortions in the images in the set of images; and rectification of the images in the set of images. ### Claim 18 (depends on 17) The method of claim 17, wherein normalizing the set of images further comprises: resampling the images to increase the similarity of corresponding pixels by correcting for geometric lens distortion; and applying scene-independent geometric corrections to the images at a sub-pixel resolution. ### Claim 19 An array camera, comprising: a plurality of cameras; a processing system; memory containing an image processing application; wherein the processor is configured by the image processing application to estimate distances to objects within a scene by: selecting a reference image from a reference viewpoint relative to the viewpoints of the set of images captured from different viewpoints; determining depth estimates for pixel locations in the reference image using at least a subset of the set of images, where a depth estimate for a given pixel location in the reference image is determined by: dividing the reference image into spatial blocks; generating depth measurements for a sparser subset of indicator pixels within the spatial blocks, where the spatial block includes a plurality of indicator pixels; assigning a depth to the pixels in each of the spatial blocks based upon the depths of the indicator pixels.