# US 2019/0356873 A1 · Image signal processing for reducing lens flare

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0356873 A1
- Document ID: US20190356873A1
- Status: Published application
- Kind: A1
- Application number: 15/984,837
- Filed: 2018-05-21
- Published: 2019-11-21
- Earliest claimed: 2018-05-21
- Assignees: GoPro, Inc.
- Applicants: GoPro, Inc.
- Inventors: Bruno César Douady-Pleven; Nicolas Boehrer; Thomas Nicolas Emmanuel Veit
- Web page: https://patentgrep.com/patent/US20190356873A1

## CPC

- G06T 5/80: Geometric correction
- H04N 25/61: the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- H10F 39/8057: {Optical shielding}
- G06T 2207/10012: Stereo images
- G06T 2207/20221: Image fusion; Image merging
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- H10F: INORGANIC SEMICONDUCTOR DEVICES SENSITIVE TO INFRARED RADIATION, LIGHT, ELECTROMAGNETIC RADIATION OF SHORTER WAVELENGTH OR CORPUSCULAR RADIATION

## Abstract

Image signal processing may include flare reduction, which may include obtaining a first input frame captured by a first image capture device of an image capture apparatus, the first image capture device having a first field-of-view, and the first input frame including lens flare corresponding to a primary light source; obtaining a second input frame captured by a second image capture device of the image capture apparatus, the second image capture device having a second field-of-view partially overlapping the first field-of-view; obtaining primary light source information corresponding to the primary light source based on the first input frame and the second input frame; obtaining a processed frame by modifying the first input frame based on the primary light source information to minimize the lens flare; and outputting the processed frame.

## Claims

### Claim 1

A non-transitory computer-readable storage medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising:
obtaining a first input frame captured by a first image capture device of an image capture apparatus, the first image capture device having a first field-of-view, and the first input frame including lens flare corresponding to a primary light source;
obtaining a second input frame captured by a second image capture device of the image capture apparatus, the second image capture device having a second field-of-view partially overlapping the first field-of-view;
obtaining primary light source information corresponding to the primary light source based on the first input frame and the second input frame;
obtaining a processed frame by modifying the first input frame based on the primary light source information to minimize the lens flare; and
outputting the processed frame.

### Claim 2 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein modifying the first input frame to minimize the lens flare includes modifying the first input frame to minimize a diffuse lens flare corresponding to the primary light source.

### Claim 3 (depends on 2)

The non-transitory computer-readable storage medium of claim 2, wherein the second input frame omits the diffuse lens flare.

### Claim 4 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein modifying the first input frame to minimize the lens flare includes modifying the first input frame to minimize a lens flare artifact.

### Claim 5 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein obtaining the primary light source information includes obtaining a difference between a first portion of the first input frame and a second portion of the second input frame.

### Claim 6 (depends on 5)

The non-transitory computer-readable storage medium of claim 5, wherein obtaining the difference between the first portion of the first input frame and the second portion of the second input frame includes:
obtaining a value of a first pixel from the first portion of the first input frame;
obtaining a value of a second pixel from the second portion of the second input frame, wherein the first pixel is spatially aligned with the second pixel according to a predicted alignment; and
obtaining a difference between the first pixel and the second pixel.

### Claim 7 (depends on 5)

The non-transitory computer-readable storage medium of claim 5, wherein obtaining the primary light source information includes:
obtaining the primary light source information by evaluating a flare model based on the difference between the first portion of the first input frame and the second portion of the second input frame.

### Claim 8 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein modifying the first input frame includes:
applying a flare model to the first input frame based on the primary light source information.

### Claim 9 (depends on 8)

The non-transitory computer-readable storage medium of claim 8, wherein obtaining the primary light source information includes obtaining primary light source orientation information and obtaining primary light source brightness information, and wherein applying the flare model to the first input frame based on the primary light source information includes applying the flare model to the first input frame based on the primary light source orientation information and the primary light source brightness information.

### Claim 10 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein obtaining the processed frame includes:
generating a combined frame based on the processed frame and the second input frame; and
using the combined frame as the processed frame.

### Claim 11 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the primary light source is the sun.

### Claim 12

A method comprising:
obtaining a first input frame captured by a first image capture device of an image capture apparatus, the first image capture device having a first field-of-view, and the first input frame including lens flare corresponding to a primary light source, wherein the first input frame includes a first image circle area based on the first field-of-view;
obtaining a second input frame captured by a second image capture device of the image capture apparatus, the second image capture device having a second field-of-view partially overlapping the first field-of-view, wherein the second input frame includes a second image circle area based on the second field-of-view;
obtaining primary light source information corresponding to the primary light source based on the first input frame and the second input frame;
obtaining a dark corner profile based on a relative illumination of an area outside the first image circle and the second image circle;
obtaining an intensity difference between the primary light source information and the dark corner profile;
obtaining a flare model;
obtaining a flare mask based on a flare difference estimation along a stitch line between the first input frame and the second input frame;
obtaining a processed frame by modifying the first input frame based on the primary light source information to minimize the lens flare, wherein modifying the first input frame includes applying the flare model to the first input frame based on the primary light source information and subtracting the flare mask; and
outputting the processed frame.

### Claim 13 (depends on 12)

The method of claim 12, wherein obtaining the flare model includes:
obtaining a difference of intensities along the stitch line between the first input frame and the second input frame; and
combining the difference of intensities with the dark corner profile.

### Claim 14 (depends on 12)

The method of claim 12, further comprising:
mapping the flare mask onto each image circle; and
diffusing the flare mask towards the center of each image.

### Claim 15

An apparatus comprising:
an image signal processor, the image signal processor configured to:
obtain a first input frame captured by a first image capture device of the apparatus, the first image capture device having a first field-of-view, and the first input frame including lens flare corresponding to a primary light source;
obtain a second input frame captured by a second image capture device of the apparatus, the second image capture device having a second field-of-view partially overlapping the first field-of-view;
obtain primary light source information corresponding to the primary light source based on the first input frame and the second input frame, wherein, to obtain the primary light source information, the image signal processor is configured to:
obtain a value of a first pixel from a first portion of the first input frame;
obtain a value of a second pixel from a second portion of the second input frame, wherein the first pixel is spatially aligned with the second pixel according to a predicted alignment;
obtain a difference between the first pixel and the second pixel; and
include, in the primary light source information, a result of evaluating a flare model based on the difference;
obtain a processed frame, wherein:
to obtain the processed frame, the image signal processor is configured to modify the first input frame based on the primary light source information to minimize the lens flare; and
to modify the first input frame, the image signal processor is configured to apply the flare model to the first input frame based on the primary light source information; and
output the processed frame.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein the image processor is further configured to:
obtain a dark corner profile based on a relative illumination of an area outside a first image circle of the first input frame and a second image circle of the second input frame; and
obtain an intensity difference between the primary light source information and the dark corner profile.

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein the image processor is further configured to:
obtain a difference of intensities along the stitch line between the first image frame and the second image frame; and
combine the difference of intensities with the dark corner profile.

### Claim 18 (depends on 16)

The apparatus of claim 16, wherein the image processor is further configured to:
obtain a flare mask based on a flare difference estimation along a stitch line between the first input frame and the second input frame.

### Claim 19 (depends on 18)

The apparatus of claim 18, wherein the image processor is further configured to:
obtain the processed frame by applying the flare model to the first input frame based on the primary light source information and subtracting the flare mask.

### Claim 20 (depends on 18)

The apparatus of claim 18, wherein the image processor is further configured to:
map the flare mask onto each image circle; and
diffuse the flare mask towards a center of each image.
