# US 2020/0396370 A1 · Apparatus and method for high dynamic range (hdr) image creation of dynamic scenes using graph cut-based labeling

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0396370 A1
- Document ID: US20200396370A1
- Status: Published application
- Kind: A1
- Application number: 16/544,008
- Filed: 2019-08-19
- Published: 2020-12-17
- Earliest claimed: 2019-06-11
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Ruiwen Zhen; John W. Glotzbach; Hamid R. Sheikh; Ibrahim Pekkucuksen; Zhen Tong; Long N. Le
- Web page: https://patentgrep.com/patent/US20200396370A1

## CPC

- H04N 23/741: by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
- G06T 5/40: using histogram techniques
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/92: based on global image properties
- H04N 23/6812: {based on additional sensors, e.g. acceleration sensors}
- H04N 23/683: {performed by a processor, e.g. controlling the readout of an image memory}
- H04N 23/71: Circuitry for evaluating the brightness variation
- G06T 2207/10144: Varying exposure
- G06T 2207/20072: Graph-based image processing
- G06T 2207/20208: High dynamic range [HDR] image processing
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method includes obtaining multiple image frames of a scene using at least one sensor of an electronic device. The multiple image frames include a first image frame and a second image frame having a longer exposure than the first image frame. The method also includes generating a label map that identifies pixels in the multiple image frames that are to be used in an image. The method further includes generating the image of the scene using the pixels extracted from the image frames based on the label map. The label map may include multiple labels, and each label may be associated with at least one corresponding pixel and may include a discrete value that identifies one of the multiple image frames from which the at least one corresponding pixel is extracted.

## Claims

### Claim 1

A method comprising:
obtaining multiple image frames of a scene using at least one sensor of an electronic device, the multiple image frames including a first image frame and a second image frame having a longer exposure than the first image frame;
generating a label map that identifies pixels in the multiple image frames that are to be used in an image; and
generating the image of the scene using the pixels extracted from the image frames based on the label map.

### Claim 2 (depends on 1)

The method of claim 1, wherein:
the label map comprises multiple labels; and
each label is associated with at least one corresponding pixel and comprises a discrete value that identifies one of the multiple image frames from which the at least one corresponding pixel is extracted.

### Claim 3 (depends on 1)

The method of claim 1, wherein:
generating the label map comprises minimizing an overall cost function that is based on a data cost function and a smoothness cost function;
the data cost function jointly considers motion and well-exposedness; and
the smoothness cost function considers how each pixel's neighbors are labeled.

### Claim 4 (depends on 3)

The method of claim 3, wherein generating the label map comprises performing a graph-cut optimization to identify labels in the label map that minimize the overall cost function.

### Claim 5 (depends on 1)

The method of claim 1, wherein:
the multiple image frames further include a third image frame having a shorter exposure than the second image frame;
part of the first image frame corresponding to part of a saturated region of the second image frame is occluded by at least one moving object in the first image frame; and
generating the image of the scene comprises recovering image details in the part of the saturated region of the second image frame using a corresponding part of the third image frame.

### Claim 6 (depends on 5)

The method of claim 5, wherein generating the image of the scene comprises:
combining the first and third image frames to produce a composite image frame using a first label map; and
combining the second image frame and the composite image frame using a second label map.

### Claim 7 (depends on 5)

The method of claim 5, wherein generating the image of the scene comprises combining the first, second, and third image frames using a three-way label map.

### Claim 8 (depends on 5)

The method of claim 5, wherein generating the image of the scene comprises:
combining the first and second image frames to produce a composite image frame using a first label map; and
combining the composite image frame and the third image frame using a second label map.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
aligning multiple captured image frames to produce at least first and second aligned image frames;
processing the first aligned image frame using histogram matching to generate a histogram-matched first aligned image frame; and
processing the first aligned image frame using tone mapping to generate a tone-matched first aligned image frame;
wherein generating the label map comprises generating the label map using the histogram-matched first aligned image frame and the second aligned image frame; and
wherein generating the image of the scene comprises using the tone-matched first aligned image frame, the second aligned image frame, and the label map.

### Claim 10

An electronic device comprising:
at least one sensor; and
at least one processing device configured to:
obtain multiple image frames of a scene using the at least one sensor, the multiple image frames including a first image frame and a second image frame having a longer exposure than the first image frame;
generate a label map that identifies pixels in the multiple image frames that are to be used in an image; and
generate the image of the scene using the pixels extracted from the image frames based on the label map.

### Claim 11 (depends on 10)

The electronic device of claim 10, wherein the at least one sensor comprises at least one camera.

### Claim 12 (depends on 10)

The apparatus of claim 10, wherein:
to generate the label map, the at least one processing device is configured to minimize an overall cost function that is based on a data cost function and a smoothness cost function;
the data cost function jointly considers motion and well-exposedness; and
the smoothness cost function considers how each pixel's neighbors are labeled.

### Claim 13 (depends on 12)

The apparatus of claim 12, wherein, to generate the label map, the at least one processing device is configured to perform a graph-cut optimization to identify labels in the label map that minimize the overall cost function.

### Claim 14 (depends on 10)

The apparatus of claim 10, wherein:
the multiple image frames further include a third image frame having a shorter exposure than the second image frame;
part of the first image frame corresponding to part of a saturated region of the second image frame is occluded by at least one moving object in the first image frame; and
to generate the image of the scene, the at least one processing device is configured to recover image details in the part of the saturated region of the second image frame using a corresponding part of the third image frame.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein, to generate the image of the scene, the at least one processing device is configured to:
combine the first and third image frames to produce a composite image frame using a first label map; and
combine the second image frame and the composite image frame using a second label map.

### Claim 16 (depends on 14)

The apparatus of claim 14, wherein, to generate the image of the scene, the at least one processing device is configured to combine the first, second, and third image frames using a three-way label map.

### Claim 17 (depends on 14)

The apparatus of claim 14, wherein, to generate the image of the scene, the at least one processing device is configured to:
combine the first and second image frames to produce a composite image frame using a first label map; and
combine the composite image frame and the third image frame using a second label map.

### Claim 18

A non-transitory machine-readable medium containing instructions that when executed cause at least one processor of an electronic device to:
obtain multiple image frames of a scene using at least one sensor of the electronic device, the multiple image frames including a first image frame and a second image frame having a longer exposure than the first image frame;
generate a label map that identifies pixels in the multiple image frames that are to be used in an image; and
generate the image of the scene using the pixels extracted from the image frames based on the label map.

### Claim 19 (depends on 18)

The non-transitory machine-readable medium of claim 18, wherein the instructions that when executed cause the at least one processor to generate the image of the scene comprise:
instructions that when executed cause the at least one processor to recover image details in a saturated region of the second image frame using a corresponding portion of the first image frame.

### Claim 20 (depends on 18)

The non-transitory machine-readable medium of claim 18, wherein:
the instructions that when executed cause the at least one processor to generate the label map comprise:
instructions that when executed cause the at least one processor to minimize an overall cost function that is based on a data cost function and a smoothness cost function;
the data cost function jointly considers motion and well-exposedness; and
the smoothness cost function considers how each pixel's neighbors are labeled.

### Claim 21 (depends on 20)

The non-transitory machine-readable medium of claim 20, wherein the instructions that when executed cause the at least one processor to generate the label map comprise:
instructions that when executed cause the at least one processor to perform a graph-cut optimization to identify labels in the label map that minimize the overall cost function.

### Claim 22 (depends on 18)

The non-transitory machine-readable medium of claim 18, wherein:
the multiple image frames further include a third image frame having a shorter exposure than the second image frame;
part of the first image frame corresponding to part of a saturated region of the second image frame is occluded by at least one moving object in the first image frame; and
the instructions that when executed cause the at least one processor to generate the image of the scene comprise:
instructions that when executed cause the at least one processor to recover image details in the part of the saturated region of the second image frame using a corresponding part of the third image frame.

### Claim 23 (depends on 22)

The non-transitory machine-readable medium of claim 22, wherein the instructions that when executed cause the at least one processor to generate the image of the scene comprise:
instructions that when executed cause the at least one processor to:
combine the first and third image frames to produce a composite image frame using a first label map; and
combine the second image frame and the composite image frame using a second label map.

### Claim 24 (depends on 22)

The non-transitory machine-readable medium of claim 22, wherein the instructions that when executed cause the at least one processor to generate the image of the scene comprise:
instructions that when executed cause the at least one processor to combine the first, second, and third image frames using a three-way label map.

### Claim 25 (depends on 22)

The non-transitory machine-readable medium of claim 22, wherein the instructions that when executed cause the at least one processor to generate the image of the scene comprise:
instructions that when executed cause the at least one processor to:
combine the first and second image frames to produce a composite image frame using a first label map; and
combine the composite image frame and the third image frame using a second label map.
