Back to results
US 9,792,668 B2Granted
Photographic image acquistion device and method
AssigneeEntropix, Inc.
InventorMikhail Korkin
Life of this patent
Estimating the term…
Abstract
An image processing device receives a multi-spectral image and a panchromatic image of a scene. The device extracts a luminosity subcomponent image from the multi-spectral image and upsamples it to generate a luminosity image of a scale intended for a super-resolved image. For each pixel of the luminosity image, the device performs a series of pixel processing and replacement steps, including extracting a first image patch surrounding the pixel and matching it with a plurality of extracted panchromatic image patches, which are smaller than the first image patch by a ratio of a size of the panchromatic image to a size of the luminosity image. The image processing and replacement of the pixels may be iteratively performed to produce a super-resolved image.
Claims29
An apparatus comprising:
- computer storage comprising a computer-readable storage medium;
- and
- a processing device operatively coupled to the computer storage and to reconstruct spatial resolution of an image of a scene captured within a multi-spectral image and a panchromatic image, the multi-spectral image obtained from a multi-spectral image sensor having a color filter array and positioned at a first optical axis, and the panchromatic image obtained from a panchromatic sensor positioned at a second optical axis that is substantially parallel to the first optical axis;
- wherein, to reconstruct the spatial resolution of the image, the processing device is to:
- extract a luminosity subcomponent image from the multi-spectral image;
- upsample the luminosity subcomponent image to an intended scale of a super-resolved image to be produced, to generate a luminosity image;
- for each pixel of a plurality of pixels of the luminosity image, to produce the super-resolved image:
- extract a first image patch surrounding the pixel;
- extract a plurality of second image patches from the panchromatic image such that the plurality of second image patches are each smaller than the first image patch by a ratio of a size of the panchromatic image to a size of the luminosity image;
- upsample the plurality of second image patches to a scale of the luminosity image, to generate a plurality of upsampled second image patches;
- match each patch of the plurality of upsampled second image patches to the first image patch;
- and
- replace a value of the pixel of the luminosity image with a new pixel value taken from a central pixel of an upsampled second image patch of the plurality of upsampled second images patches that most closely matches the first image patch;
- and
- store, in the computer storage, the super-resolved image as a replacement of the image.
12 more elements
A method comprising:
- receiving, from an image acquisition device by an image processing device, a multi-spectral image and a panchromatic image both taken of a scene, the multi-spectral image obtained from a multi-spectral image sensor having a color filter array and positioned at a first optical axis, and the panchromatic image obtained from a panchromatic sensor positioned at a second optical axis that converges at an angle with the first optical axis;
- extracting, by the image processing device, a luminosity subcomponent image from the multi-spectral image;
- upsampling, by the image processing device, the luminosity subcomponent image to an intended scale of a super-resolved image to be produced, to generate a luminosity image;
- for each pixel of a plurality of pixels of the luminosity image, to produce the super-resolved image, the image processing device:
- extracting a first image patch surrounding the pixel;
- extracting a plurality of second image patches from the panchromatic image such that the plurality of second image patches are each smaller than the first image patch by a ratio of a size of the panchromatic image to a size of the luminosity image;
- downsampling the first image patch to a scale of the plurality of second image patches, to generate a downsampled first image patch;
- matching each patch of the plurality of second image patches to the downsampled first image patch;
- and
- replacing a value of the pixel of the luminosity image with a new pixel value taken from a central pixel of a second image patch of the plurality of second image patches that most closely matches the downsampled first image patch;
- and
- storing, in a computer storage of the image processing device, the super-resolved image as a replacement to the luminosity image.
9 more elements
A non-transitory, computer-readable medium storing instructions, which when executed by an image processing device, cause the image processing device to perform a plurality of operations comprising:
- receiving, from a multi-aperture image acquisition device by the image processing device, a first image of a scene projected through a first aperture onto a first digital image sensor and a second image of the scene projected through one or more second aperture onto one or more second digital image sensor;
- fusing the second image with the first image, by the image processing device, to generate an output image that compensates for parallax between the first image and the second image, wherein fusing comprises, for each pixel of a subcomponent image of the first image:
- extracting a first image patch surrounding the pixel;
- extracting a plurality of second image patches from the second image;
- matching each of the plurality of second image patches to the first image patch;
- and
- replacing a value of the pixel with a new pixel value taken from a central pixel of a second image patch of the plurality of second image patches that most closely matches the first image patch;
- and
- storing, in a computer storage of the image processing device, the output image as a replacement to the first image.
6 more elements
Dependencies are inferred from the claim text.