# US 2017/0256036 A1 · Automatic microlens array artifact correction for light-field images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0256036 A1
- Document ID: US20170256036A1
- Status: Published application
- Kind: A1
- Application number: 15/059,657
- Filed: 2016-03-03
- Published: 2017-09-07
- Earliest claimed: 2016-03-03
- Assignees: Lytro, Inc.
- Applicants: Lytro, Inc.
- Inventors: Alex Song; Yuriy Romanenko
- Web page: https://patentgrep.com/patent/US20170256036A1

## CPC

- H04N 23/60: Control of cameras or camera modules
- G02B 3/0006: {Arrays (G02B3/02, G02B5/188 take precedence)}
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/77: Retouching; Inpainting; Scratch removal
- H04N 23/80: Camera processing pipelines; Components thereof
- H04N 23/957: Light-field or plenoptic cameras or camera modules
- H04N 23/958: for extended depth of field imaging
- G06T 2207/10152: Varying illumination
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

According to various embodiments, light-field image data may be processed so as to prevent, remove, and/or mitigate artifacts caused by previous processing steps. A light-field image may first be captured by a light-field camera. One or more processing steps may be applied to the light-field image to generate a processed light-field image having one or more artifacts. A flat white modulation light-field image may also be captured by the light-field camera. The same processing steps previously applied to the light-field image may be applied to the modulation light-field image to generate a processed modulation light-field image with the same artifacts. The artifacts may then be identified in the processed modulation light-field image to generate an identification of the artifacts. This identification may be used to identify the same artifacts in the processed light-field image, which may then be corrected to remove the artifacts to generate a corrected, processed light-field image.

## Claims

### Claim 1

A method for correcting a light-field image to remove artifacts, the method comprising:
in a data store, receiving the light-field image after the light-field image has been captured by a light-field image capture device having an image sensor and a microlens array;
in a processor, applying one or more processing steps to the light-field image to generate a processed light-field image comprising one or more artifacts;
in the data store, receiving a modulation light-field image that depicts no significant features;
in the processor, applying the one or more processing steps to the modulation light-field image to generate a processed modulation light-field image comprising the artifacts;
in the processor, identifying the artifacts in the processed modulation light-field image to generate an identification of the artifacts;
in the processor, using the identification of the artifacts to identify the artifacts in the processed light-field image; and
in the processor, correcting the processed light-field image to remove the artifacts to generate a corrected, processed light-field image.

### Claim 2 (depends on 1)

The method of claim 1, wherein the modulation light-field image comprises a light-field image of a uniform flat white scene.

### Claim 3 (depends on 2)

The method of claim 2, wherein:
the light-field image comprises a light-field image captured by the light-field image capture device using capture settings; and
the modulation light-field image comprises a modulation light-field image captured by the light-field image capture device using modulation capture settings that are approximately the same as the capture settings.

### Claim 4 (depends on 3)

The method of claim 3, further comprising, in the light-field image capture device:
capturing the light-field image using the capture settings; and
capturing the modulation light-field image using the modulation capture settings.

### Claim 5 (depends on 4)

The method of claim 4, wherein the one or more artifacts are caused by aliasing of a pattern of microlenses in the microlens array.

### Claim 6 (depends on 4)

The method of claim 4, wherein the capture settings comprise a zoom setting of the light-field image capture device.

### Claim 7 (depends on 6)

The method of claim 6, wherein the capture settings further comprise a focus setting of the light-field image capture device.

### Claim 8 (depends on 1)

The method of claim 1, wherein:
applying the one or more processing steps to the light-field image comprises downsampling the light-field image; and
applying the one or more processing steps to the modulation light-field image comprises downsampling the modulation light-field image.

### Claim 9 (depends on 1)

The method of claim 1, wherein:
applying the one or more processing steps to the light-field image comprises applying a filter to the light-field image; and
applying the one or more processing steps to the modulation light-field image comprises applying the filter to the modulation light-field image.

### Claim 10 (depends on 1)

The method of claim 1, wherein identifying the artifacts in the processed modulation light-field image comprises applying an autofocus edge detection algorithm to the processed modulation light-field image.

### Claim 11

A non-transitory computer-readable medium for correcting a light-field image to remove artifacts, comprising instructions stored thereon, that when executed by a processor, perform the steps of:
causing a data store to receive the light-field image after the light-field image has been captured by a light-field image capture device having an image sensor and a microlens array;
applying one or more processing steps to the light-field image to generate a processed light-field image comprising one or more artifacts;
causing the data store to receive a modulation light-field image that depicts no significant features;
applying the one or more processing steps to the modulation light-field image to generate a processed modulation light-field image comprising the artifacts;
identifying the artifacts in the processed modulation light-field image to generate an identification of the artifacts;
using the identification of the artifacts to identify the artifacts in the processed light-field image; and
correcting the processed light-field image to remove the artifacts to generate a corrected, processed light-field image.

### Claim 12 (depends on 11)

The non-transitory computer-readable medium of claim 11, wherein the modulation light-field image comprises a light-field image of a uniform flat white scene.

### Claim 13 (depends on 12)

The non-transitory computer-readable medium of claim 12, wherein:
the light-field image comprises a light-field image captured by the light-field image capture device using capture settings; and
the modulation light-field image comprises a modulation light-field image captured by the light-field image capture device using modulation capture settings that are approximately the same as the capture settings.

### Claim 14 (depends on 13)

The non-transitory computer-readable medium of claim 13, wherein the one or more artifacts are caused by aliasing of a pattern of microlenses in the microlens array.

### Claim 15 (depends on 13)

The non-transitory computer-readable medium of claim 13, wherein the capture settings comprise at least one selected from the group consisting of:
a zoom setting of the light-field image capture device; and
a focus setting of the light-field image capture device.

### Claim 16 (depends on 11)

The non-transitory computer-readable medium of claim 11, wherein:
applying the one or more processing steps to the light-field image comprises downsampling the light-field image; and
applying the one or more processing steps to the modulation light-field image comprises downsampling the modulation light-field image.

### Claim 17 (depends on 11)

The non-transitory computer-readable medium of claim 11, wherein:
applying the one or more processing steps to the light-field image comprises applying a filter to the light-field image; and
applying the one or more processing steps to the modulation light-field image comprises applying the filter to the modulation light-field image.

### Claim 18 (depends on 11)

The non-transitory computer-readable medium of claim 11, wherein identifying the artifacts in the processed modulation light-field image comprises applying an autofocus edge detection algorithm to the processed modulation light-field image.

### Claim 19

A system for correcting a light-field image to remove artifacts, the system comprising:
a data store configured to receive the light-field image after the light-field image has been captured by a light-field image capture device having an image sensor and a microlens array;
a processor, communicatively coupled to the data store, configured to applying one or more processing steps to the light-field image to generate a processed light-field image comprising one or more artifacts;
wherein the data store is further configured to receive a modulation light-field image that depicts no significant features;
and wherein the processor is further configured to:
apply the one or more processing steps to the modulation light-field image to generate a processed modulation light-field image comprising the artifacts;
identify the artifacts in the processed modulation light-field image to generate an identification of the artifacts;
use the identification of the artifacts to identify the artifacts in the processed light-field image; and
correct the processed light-field image to remove the artifacts to generate a corrected, processed light-field image.

### Claim 20 (depends on 19)

The system of claim 19, wherein the modulation light-field image comprises a light-field image of a uniform flat white scene.

### Claim 21 (depends on 20)

The system of claim 20, further comprising the light-field image capture device, wherein:
the light-field image capture device is configured to capture the light-field image with capture settings applied;
the modulation light-field image was captured with modulation capture settings applied; and
the modulation capture settings are approximately the same as the capture settings.

### Claim 22 (depends on 21)

The system of claim 21, wherein the one or more artifacts are caused by aliasing of a pattern of microlenses in the microlens array.

### Claim 23 (depends on 21)

The system of claim 21, wherein the capture settings comprise at least one selected from the group consisting of:
a zoom setting of the light-field image capture device; and
a focus setting of the light-field image capture device.

### Claim 24 (depends on 19)

The system of claim 19, wherein:
the processor is further configured to apply the one or more processing steps to the light-field image by downsampling the light-field image; and
the processor is further configured to apply the one or more processing steps to the modulation light-field image by downsampling the modulation light-field image.

### Claim 25 (depends on 19)

The system of claim 19, wherein:
the processor is further configured to apply the one or more processing steps to the light-field image by applying a filter to the light-field image; and
the processor is further configured to apply the one or more processing steps to the modulation light-field image by applying the filter to the modulation light-field image.

### Claim 26 (depends on 19)

The system of claim 19, wherein the processor is further configured to identify the artifacts in the processed modulation light-field image by applying an autofocus edge detection algorithm to the processed modulation light-field image.
