# US 10,375,302 B2 · Systems and methods for controlling aliasing in images captured by an array camera for use in super resolution processing using pixel apertures > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,375,302 B2 - Document ID: US10375302B2 - Status: Granted - Kind: B2 - Application number: 15/785,315 - Filed: 2017-10-16 - Published: 2018-04-05 - Granted: 2019-08-06 - Earliest claimed: 2011-09-19 - Assignees: FotoNation Limited - Applicants: FotoNation Limited - Inventors: Shree Nayar; Kartik Venkataraman; Bedabrata Pain; Dan Lelescu - Web page: https://patentgrep.com/patent/US10375302B2 ## CPC - H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio - H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps - H04N 23/16: Optical arrangements associated therewith, e.g. for beam-splitting or for colour correction - H04N 23/45: for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images - H04N 25/41: {Extracting pixel data from a plurality of image sensors simultaneously picking up an image, e.g. for increasing the field of view by combining the outputs of a plurality of sensors} - H04N 13/232: using fly-eye lenses, e.g. arrangements of circular lenses - H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Imager arrays, array camera modules, and array cameras in accordance with embodiments of the invention utilize pixel apertures to control the amount of aliasing present in captured images of a scene. One embodiment includes a plurality of focal planes, control circuitry configured to control the capture of image information by the pixels within the focal planes, and sampling circuitry configured to convert pixel outputs into digital pixel data. In addition, the pixels in the plurality of focal planes include a pixel stack including a microlens and an active area, where light incident on the surface of the microlens is focused onto the active area by the microlens and the active area samples the incident light to capture image information, and the pixel stack defines a pixel area and includes a pixel aperture, where the size of the pixel apertures is smaller than the pixel area. ## Claims ### Claim 1 An array camera, comprising: a plurality of cameras; a processor configured to receive digital pixel data from the plurality of cameras via interface circuitry; and memory containing an image processing pipeline application; wherein the image processing pipeline application directs the processor to: obtain a set of images of a scene that include aliasing from the plurality of cameras, wherein the aliasing is different in each image; fuse portions of the set of images to form fused image at each of a plurality of hypothesized disparities; compare the portion of the fused image obtained at each hypothesized disparity to the scene captured in the set of images; select a hypothesized disparity at which the portion of the fused image is most similar to the scene captured in the set of images accounting for aliasing as a disparity estimate; and synthesize an image of the scene from a reference viewpoint using the set of images and disparity information including the disparity estimate. ### Claim 2 (depends on 1) The array camera of claim 1, wherein the image processing pipeline application configures the processor to compare the portion of the fused image to the scene captured in the set of images by determining the similarity of pixels in pixel stacks in the portion of the fused image. ### Claim 3 (depends on 1) The array camera of claim 1, wherein the image processing pipeline application directs the processor to: compare portions of fused images formed by fusing at least two subsets of the set of images at a plurality of hypothesized disparities; and select the hypothesized disparity at which the compared portions of fused higher resolution images are most similar. ### Claim 4 (depends on 3) The array camera of claim 3, wherein at least one image in the set of images is common to the at least two subsets of the set of images. ### Claim 5 (depends on 4) The array camera of claim 4, wherein the at least two subsets of the set of images include a common image having a viewpoint that is the reference viewpoint. ### Claim 6 (depends on 5) The array camera of claim 5, wherein the at least two subsets of the set of images include common images captured from viewpoints above, below, to the left, and to the right of the reference viewpoint. ### Claim 7 (depends on 3) The array camera of claim 3, wherein the reference viewpoint is the viewpoint of one of the images in the set of images. ### Claim 8 (depends on 3) The array camera of claim 3, wherein the image processing pipeline application configures the processor to fuse at least two subsets of the set of images at a hypothesized disparity by: identifying pixels within the subset of the set of images based upon the hypothesized disparity and the viewpoints of the images; fusing the identified pixels onto a grid generated from a chosen reference viewpoint using known calibration information; and performing hole filing to fill holes in locations in the grid. ### Claim 9 (depends on 3) The array camera of claim 3, wherein the image processing pipeline application configures the processor to compare portions of fused images by comparing matching error of the portions of the fused images. ### Claim 10 (depends on 9) The array camera of claim 9, wherein matching error is determined using at least one selected from the group of the L1-norm and the L2-norm of the difference of the portions of the first and second image. ### Claim 11 (depends on 1) The array camera of claim 1, wherein: the plurality of cameras is arranged as an imager array comprising: a plurality of focal planes, where each focal plane comprises a two dimensional arrangement of pixels having at least two pixels in each dimension and each focal plane is contained within a region of the imager array that does not contain pixels from another focal plane; wherein the pixels in the plurality of focal planes comprise a pixel stack including a microlens and an active area, where light incident on the surface of the microlens is focused onto the active area by the microlens and the active area samples the incident light to capture image information; and wherein the pixel stack defines a pixel area and includes a pixel aperture, where the size of the pixel apertures is smaller than the pixel area. ### Claim 12 (depends on 11) The array camera of claim 11, wherein the pixel aperture is formed by a microlens that is smaller than the pixel area. ### Claim 13 (depends on 12) The array camera of claim 12, wherein gaps exist between adjacent microlenses in the pixel stacks of adjacent pixels in a focal plane. ### Claim 14 (depends on 13) The array camera of claim 13, wherein light is prevented from entering the pixel stacks through the gaps between the microlenses by a light blocking material. ### Claim 15 (depends on 11) The array camera of claim 11, wherein the pixel stack includes a color filter. ### Claim 16 (depends on 15) The array camera of claim 15, wherein the color filters in the pixel stacks of the arrangement of pixels within a focal plane are the same. ### Claim 17 (depends on 11) The array camera module of claim 11, wherein the plurality of cameras further comprises: an optic array of lens stacks, where an image including aliasing is formed on each focal plane by a separate lens stack in the optic array of lens stacks; and at least one of the plurality of lens stacks includes a color filter and the pixel stacks of the arrangement of pixels within the focal plane on which said at least one of the plurality of lens stacks forms an image do not include color filters. ### Claim 18 (depends on 11) The array camera module of claim 11, wherein the pixel aperture is formed using at least one light blocking material. ### Claim 19 A method of determining a disparity of a point in a scene using image data comprising a set of images that capture the scene, the method comprising: capturing a set of images, where each image includes aliasing and the aliasing is different in each captured image; fusing portions of the set of images to form a portion of an image at each of a plurality of hypothesized disparities; comparing the portion of the fused higher resolution image obtained at each hypothesized disparity to the scene captured in the set of low resolution images; and selecting the hypothesized disparity at which the portion of the fused image is most similar to the scene captured in the set of images as the disparity of at least one point in the scene captured by the set of low resolution images. ### Claim 20 A method of determining a disparity of a point in a scene using image data comprising a set of images that capture the scene, the method comprising: fusing portions of a first subset of a set of images to form a portion of a first image at each of a plurality of hypothesized disparities; fusing portions of a second subset of the set of images to form a portion of a second image at each of the plurality of hypothesized disparities; comparing at least the portions of the first and second images fused at each of the plurality of hypothesized disparities, wherein comparing at least the portions of the first and second images fused at each of the plurality of hypothesized disparities comprises comparing the portions of the first and second images for matching error at each of the plurality of hypothesized disparities; and selecting the hypothesized disparity at which the portions of the compared images are most similar as the disparity of at least one point in the scene imaged by pixels within portions of the first and second subsets of the set of images.