# US 2016/0241797 A1 · Devices, systems, and methods for single-shot high-resolution multispectral image acquisition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0241797 A1
- Document ID: US20160241797A1
- Status: Published application
- Kind: A1
- Application number: 14/962,486
- Filed: 2015-12-08
- Published: 2016-08-18
- Earliest claimed: 2015-02-17
- Assignees: Canon Kabushiki Kaisha
- Applicants: Canon Kabushiki Kaisha
- Inventors: Jinwei Ye
- Web page: https://patentgrep.com/patent/US20160241797A1

## CPC

- H04N 23/957: Light-field or plenoptic cameras or camera modules
- G02B 30/27: involving lenticular arrays
- H04N 23/10: for generating image signals from different wavelengths
- H04N 25/11: Arrangement of colour filter arrays [CFA]; Filter mosaics
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS

## Abstract

Systems, methods, and devices for generating high-resolution multispectral light-field images are described. The systems and devices a main lens include a microlens array, a multispectral-filter array that comprises spectral filters that filter light in different wavelengths, and a sensor that is configured to detect incident light. Also, the main lens, the microlens array, the multispectral-filter array, and the light sensor are disposed such that light from a scene passes through the main lens, the microlens array, and the multispectral-filter array and strikes a sensing surface of the sensor. Additionally, the multispectral-filter array is disposed so as to encode, in the light that strikes the sensing surface, a plane of the microlens array on the sensing surface of the sensor. Furthermore, the systems, methods, and devices generate high-resolution multispectral light field-images from low-resolution sub-aperture images using an optimization framework that uses a first order gradient sparsity in intensity and a second order gradient sparsity in wavelength as regularization terms.

## Claims

### Claim 1

A system comprising:
a main lens;
a microlens array;
a multispectral-filter array that comprises spectral filters that filter light in different wavelengths; and
a sensor that is configured to detect incident light,
wherein the main lens, the microlens array, the multispectral-filter array, and the light sensor are disposed such that light from a scene passes through the main lens, the microlens array, and the multispectral-filter array and strikes a sensing surface of the sensor, and
wherein the multispectral-filter array is disposed so as to encode, in the light that strikes the sensing surface, a plane of the microlens array on the sensing surface of the sensor.

### Claim 2 (depends on 1)

The system of claim 1, wherein the main lens is focused on the microlens array.

### Claim 3 (depends on 2)

The system of claim 2, wherein the microlens array is focused on the sensor.

### Claim 4 (depends on 1)

The system of claim 1, wherein the microlens array is disposed between the main lens and the multispectral-filter array.

### Claim 5 (depends on 1)

The system of claim 1, wherein the multispectral-filter array is disposed between the main lens and the microlens array.

### Claim 6 (depends on 1)

The system of claim 1, wherein a number of spectral filters in the multispectral-filter array is equal to a number of microlenses in the microlens array.

### Claim 7 (depends on 6)

The system of claim 6, wherein each microlens in the microlens array is aligned with a respective corresponding spectral filter of the multispectral-filter array.

### Claim 8 (depends on 7)

The system of claim 7, wherein the microlens array and the multispectral-filter array are disposed such that, if a photon travels from the main lens through a microlens to the sensing surface of the sensor, the photon can pass through only the corresponding spectral filter that is aligned with the microlens.

### Claim 9 (depends on 7)

The system of claim 7, wherein the microlens array and the multispectral-filter array are disposed such that, between the microlens array and the sensing surface of the sensor, rays of light that pass through a microlens and the corresponding spectral filter do not overlap with rays of light that pass through other microlenses and their corresponding spectral filters.

### Claim 10 (depends on 7)

The system of claim 7, wherein all photons that travel through a microlens and the respective spectral filter that is aligned with the microlens strike within a respective microlens-image area on the sensing surface of the sensor.

### Claim 11

A system comprising:
one or more computer-readable storage media; and
one or more processors that are coupled to the one or more computer-readable storage media and that are configured to cause the system to
obtain a multispectral image that is composed of microlens images, wherein each microlens image was captured by a respective microlens-image area of a sensor, and wherein each microlens image was generated based on light that passed through a main lens, a respective microlens of a microlens array, and a respective spectral filter of a multispectral-filter array and that was detected by the respective microlens-image area of the sensor, and
generate sub-aperture images from the microlens images, wherein a sub-aperture image includes a pixel from each microlens image.

### Claim 12 (depends on 11)

The system of claim 11, wherein each microlens image includes L×L pixels, and wherein the one or more processors are further configured to cause the system to generate L×L sub-aperture images.

### Claim 13 (depends on 11)

The system of claim 11, wherein, to generate the sub-aperture images from the microlens images, the one or more processors are configured to cause the system to assign a pixel from a microlens image to a position in a sub-aperture image that corresponds to a position of the microlens image in the captured image.

### Claim 14 (depends on 11)

The system of claim 11, wherein the captured image includes N×N microlens images, and wherein each sub-aperture image includes N×N pixels.

### Claim 15 (depends on 11)

The system of claim 11, wherein the multispectral-filter array includes a first spectral filter that is configured to selectively transmit a first spectrum of light and includes a second spectral filter that is configured to selectively transmit a second spectrum of light that is different from the first spectrum,
wherein one of the microlens images was generated from light that passed through the first spectral filter, and
wherein one of the microlens images was generated from light that passed through the second spectral filter.

### Claim 16 (depends on 11)

The system of claim 11, wherein the one or more processors are further configured to cause the system to generate the sub-aperture images from the microlens images based on sub-pixel shifts and spectral filtering according to a downsample operation.

### Claim 17 (depends on 16)

The system of claim 16, wherein the sub-pixel shifts are computed using a depth of a scene that is depicted in the multispectral image.

### Claim 18 (depends on 16)

The system of claim 16, wherein the spectral filtering uses a multispectral-filter array that is identical to a multispectral-filter array that captured the multispectral image.

### Claim 19

One or more non-transitory computer-readable media storing instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising:
obtaining sub-aperture images; and
generating a higher-resolution multispectral image from the sub-aperture images based on the sub-aperture images and on a sparsity prior in second-order gradients of spectral images in a wavelength domain.

### Claim 20 (depends on 19)

The one or more non-transitory computer-readable media of claim 19, wherein generating the higher-resolution multispectral image from the sub-aperture images uses an optimization process.

### Claim 21 (depends on 19)

The one or more non-transitory computer-readable media of claim 19, wherein generating the higher-resolution multispectral image from the sub-aperture images is further based on a sparsity prior in first-order gradients of spectral images in an intensity domain.

### Claim 22 (depends on 19)

The one or more non-transitory computer-readable media of claim 19,
wherein the sub-aperture images were generated from microlens images,
wherein a sub-aperture image include a pixel from each microlens image,
wherein each microlens image was captured by a respective microlens-image area of a sensor, and
wherein each microlens image was generated based on light that passed through a main lens, a respective microlens of a microlens array, and a respective spectral filter of a multispectral-filter array and that was detected by the respective microlens-image area of the sensor.
