# US 2024/0257325 A1 · Tone mapping in high-resolution imaging systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0257325 A1
- Document ID: US20240257325A1
- Status: Published application
- Kind: A1
- Application number: 18/485,970
- Filed: 2023-10-12
- Published: 2024-08-01
- Earliest claimed: 2023-01-26
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Abhiram Gnanasambandam; John W. Glotzbach; Zeeshan Nadir; Gunawath Dilshan Godaliyadda; John Seokjun Lee; Hamid R. Sheikh
- Web page: https://patentgrep.com/patent/US20240257325A1

## CPC

- G06T 5/94: based on local image properties, e.g. for local contrast enhancement
- G06T 3/4015: Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
- G06T 5/30: Erosion or dilatation, e.g. thinning
- G06T 5/40: using histogram techniques
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 15/506: {Illumination models}
- G06T 2207/10024: Color image
- G06T 2207/10144: Varying exposure
- G06T 2207/20012: Locally adaptive
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20208: High dynamic range [HDR] image processing
- G06T 2207/20221: Image fusion; Image merging
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method includes obtaining multiple image frames captured using at least one imaging sensor. The method also includes generating a local tone map, a global tone map look-up table (LUT), and one or more contrast enhancement LUTs based on at least one of the image frames and one or more parameters of the at least one imaging sensor. The method further includes generating a blended and demosaiced image based on the image frames and generating a local tone mapped image based on the blended and demosaiced image and the local tone map. The method also includes adjusting color saturation based on the local tone mapped image to generate a corrected image. In addition, the method includes generating an output image based on the corrected image, the global tone map LUT, and the one or more contrast enhancement LUTs.

## Claims

### Claim 1

A method comprising:
obtaining multiple image frames captured using at least one imaging sensor;
generating a local tone map, a global tone map look-up table (LUT), and one or more contrast enhancement LUTs based on at least one of the image frames and one or more parameters of the at least one imaging sensor;
generating a blended and demosaiced image based on the image frames;
generating a local tone mapped image based on the blended and demosaiced image and the local tone map;
adjusting color saturation based on the local tone mapped image to generate a corrected image; and
generating an output image based on the corrected image, the global tone map LUT, and the one or more contrast enhancement LUTs.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating the local tone map comprises:
selecting one of the image frames;
generating a gain map based on the selected image frame;
dividing the gain map into multiple tiles; and
generating a three-dimensional (3D) LUT using the tiles.

### Claim 3 (depends on 2)

The method of claim 2, wherein generating the gain map comprises:
down-sampling the selected image frame to generate a down-sampled image frame;
converting the down-sampled image frame into a single luma channel image;
applying local tone mapping to the single luma channel image to generate a tone mapped image; and
determining gain values of the gain map based on ratios of values in the tone mapped image and values in the single luma channel image.

### Claim 4 (depends on 3)

The method of claim 3, wherein generating the one or more contrast enhancement LUTs comprises generating the one or more contrast enhancement LUTs based on the tone mapped image.

### Claim 5 (depends on 2)

The method of claim 2, wherein generating the 3D LUT comprises:
determining gain values for locations of a profile gain table map (PGTM);
filling in one or more gaps of the PGTM, each gap associated with a gain value that is initially missing in the PGTM due to a lack of image data, the PGTM defining intensity-gain curves;
for one or more of the intensity-gain curves that end prematurely, concatenating each of the one or more intensity-gain curves with a parameterized tail;
filtering the intensity-gain curves in an intensity dimension and in a spatial dimension; and
for at least one of the intensity-gain curves having a parameterized tail with a gradient larger than a threshold, smoothing the parameterized tail.

### Claim 6 (depends on 1)

The method of claim 1, wherein generating the global tone map LUT comprises generating the global tone map LUT based on the one or more parameters of the at least one imaging sensor.

### Claim 7 (depends on 1)

The method of claim 1, wherein generating the local tone mapped image comprises:
applying the local tone map to the blended and demosaiced image to generate a first gain map;
performing a morphological opening operation using the first gain map to generate a second gain map;
performing a guided filtering operation using the first and second gain maps to generate a third gain map; and
generating the local tone mapped image using the blended and demosaiced image and the third gain map.

### Claim 8 (depends on 1)

The method of claim 1, wherein:
the color saturation is adjusted prior to performing a gamma correction;
the corrected image undergoes the gamma correction and is converted from a red-green-blue (RGB) image domain to a luma-chroma (YUV) image domain; and
generating the output image comprises applying contrast enhancement using the one or more contrast enhancement LUTs and applying global tone mapping using the global tone map LUT to a luma channel in the YUV image domain.

### Claim 9

An electronic device comprising:
at least one imaging sensor configured to capture multiple image frames; and
at least one processing device configured to:
generate a local tone map, a global tone map look-up table (LUT), and one or more contrast enhancement LUTs based on at least one of the image frames and one or more parameters of the at least one imaging sensor;
generate a blended and demosaiced image based on the image frames;
generate a local tone mapped image based on the blended and demosaiced image and the local tone map;
adjust color saturation based on the local tone mapped image to generate a corrected image; and
generate an output image based on the corrected image, the global tone map LUT, and the one or more contrast enhancement LUTs.

### Claim 10 (depends on 9)

The electronic device of claim 9, wherein, to generate the local tone map, the at least one processing device is configured to:
select one of the image frames;
generate a gain map based on the selected image frame;
divide the gain map into multiple tiles; and
generate a three-dimensional (3D) LUT using the tiles.

### Claim 11 (depends on 10)

The electronic device of claim 10, wherein, to generate the gain map, the at least one processing device is configured to:
down-sample the selected image frame to generate a down-sampled image frame;
convert the down-sampled image frame into a single luma channel image;
apply local tone mapping to the single luma channel image to generate a tone mapped image; and
determine gain values of the gain map based on ratios of values in the tone mapped image and values in the single luma channel image.

### Claim 12 (depends on 11)

The electronic device of claim 11, wherein the at least one processing device is configured to generate the one or more contrast enhancement LUTs based on the tone mapped image.

### Claim 13 (depends on 10)

The electronic device of claim 10, wherein, to generate the 3D LUT, the at least one processing device is configured to:
determine gain values for locations of a profile gain table map (PGTM);
fill in one or more gaps of the PGTM, each gap associated with a gain value that is initially missing in the PGTM due to a lack of image data, the PGTM defining intensity-gain curves;
for one or more of the intensity-gain curves that end prematurely, concatenate each of the one or more intensity-gain curves with a parameterized tail;
filter the intensity-gain curves in an intensity dimension and in a spatial dimension; and
for at least one of the intensity-gain curves having a parameterized tail with a gradient larger than a threshold, smooth the parameterized tail.

### Claim 14 (depends on 9)

The electronic device of claim 9, wherein the at least one processing device is configured to generate the global tone map LUT based on the one or more parameters of the at least one imaging sensor.

### Claim 15 (depends on 9)

The electronic device of claim 9, wherein, to generate the local tone mapped image, the at least one processing device is configured to:
apply the local tone map to the blended and demosaiced image to generate a first gain map;
perform a morphological opening operation using the first gain map to generate a second gain map;
perform a guided filtering operation using the first and second gain maps to generate a third gain map; and
generate the local tone mapped image using the blended and demosaiced image and the third gain map.

### Claim 16 (depends on 9)

The electronic device of claim 9, wherein:
the at least one processing device is configured to adjust the color saturation prior to performing a gamma correction;
the at least one processing device is further configured to apply the gamma correction to the corrected image and convert from a red-green-blue (RGB) image domain to a luma-chroma (YUV) image domain, and
to generate the output image, the at least one processing device is configured to apply contrast enhancement using the one or more contrast enhancement LUTs and apply global tone mapping using the global tone map LUT to a luma channel in the YUV image domain.

### Claim 17

A non-transitory machine readable medium containing instructions that when executed cause at least one processor of an electronic device to:
obtain multiple image frames captured using at least one imaging sensor;
generate a local tone map, a global tone map look-up table (LUT), and one or more contrast enhancement LUTs based on at least one of the image frames and one or more parameters of the at least one imaging sensor;
generate a blended and demosaiced image based on the image frames;
generate a local tone mapped image based on the blended and demosaiced image and the local tone map;
adjust color saturation based on the local tone mapped image to generate a corrected image; and
generate an output image based on the corrected image, the global tone map LUT, and the one or more contrast enhancement LUTs.

### Claim 18 (depends on 17)

The non-transitory machine readable medium of claim 17, wherein:
the instructions that when executed cause the at least one processor to generate the local tone map comprise instructions that when executed cause the at least one processor to:
select one of the image frames;
generate a gain map based on the selected image frame;
divide the gain map into multiple tiles; and
generate a three-dimensional (3D) LUT using the tiles;
the instructions that when executed cause the at least one processor to generate the gain map comprise instructions that when executed cause the at least one processor to:
down-sample the selected image frame to generate a down-sampled image frame;
convert the down-sampled image frame into a single luma channel image;
apply local tone mapping to the single luma channel image to generate a tone mapped image; and
determine gain values of the gain map based on ratios of values in the tone mapped image and values in the single luma channel image; and
the instructions that when executed cause the at least one processor to generate the one or more contrast enhancement LUTs comprise instructions that when executed cause the at least one processor to generate the one or more contrast enhancement LUTs based on the tone mapped image.

### Claim 19 (depends on 18)

The non-transitory machine readable medium of claim 18, wherein the instructions that when executed cause the at least one processor to generate the 3D LUT comprise instructions that when executed cause the at least one processor to:
determine gain values for locations of a profile gain table map (PGTM);
fill in one or more gaps of the PGTM, each gap associated with a gain value that is initially missing in the PGTM due to a lack of image data, the PGTM defining intensity-gain curves;
for one or more of the intensity-gain curves that end prematurely, concatenate each of the one or more intensity-gain curves with a parameterized tail;
filter the intensity-gain curves in an intensity dimension and in a spatial dimension; and
for at least one of the intensity-gain curves having a parameterized tail with a gradient larger than a threshold, smooth the parameterized tail.

### Claim 20 (depends on 17)

The non-transitory machine readable medium of claim 17, wherein the instructions that when executed cause the at least one processor to generate the local tone mapped image comprise instructions that when executed cause the at least one processor to:
apply the local tone map to the blended and demosaiced image to generate a first gain map;
perform a morphological opening operation using the first gain map to generate a second gain map;
perform a guided filtering operation using the first and second gain maps to generate a third gain map; and
generate the local tone mapped image using the blended and demosaiced image and the third gain map.
