# US 2024/0202867 A1 · Demosaicing method and demosaicing device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202867 A1
- Document ID: US20240202867A1
- Status: Published application
- Kind: A1
- Application number: 18/222,196
- Filed: 2023-07-14
- Published: 2024-06-20
- Earliest claimed: 2022-12-14
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Pilsu Kim; Seungwon Choi; Jongseong Choi; Byeongdae Lee
- Web page: https://patentgrep.com/patent/US20240202867A1

## CPC

- G06T 3/4015: Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
- G06T 5/20: using local operators
- G06T 7/13: Edge detection
- G06T 7/90: Determination of colour characteristics
- G06V 10/56: relating to colour
- G06T 2207/10024: Color image
- G06T 2207/20192: Edge enhancement; Edge preservation
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A demosaicing method for an input mosaic image includes: obtaining an input image, for each sub target pixel, estimating a main edge direction based on main color pixel values; determining the main edge direction by performing edge enhancement based on the input mosaic image; generating a main color demosaiced image by performing a main interpolation based on the main edge direction to generate interpolated main color pixel values corresponding to the sub target pixels; and generating a first sub color demosaiced image and a second sub color demosaiced image by performing a sub interpolation to generate interpolated first sub color pixel values and interpolated second sub color pixel values for the sub target pixels and for main target pixels corresponding to the main color pixel values.

## Claims

### Claim 1

A demosaicing method for an input mosaic image, the demosaicing method comprising:
obtaining an input image comprising main color pixel values corresponding to a main color, first sub color pixel values corresponding to a first sub color, and second sub color pixel values corresponding to a second sub color, wherein the main color pixel values, the first sub color pixel values, and the second sub color pixel values are arranged in a regular pattern;
for each sub target pixel from among sub target pixels corresponding to the first sub color pixel values and the second sub color pixel values, estimating a main edge direction based on the main color pixel values;
determining the main edge direction by performing edge enhancement based on the input mosaic image;
generating a main color demosaiced image corresponding to the main color by performing a main interpolation based on the main edge direction to generate interpolated main color pixel values corresponding to the sub target pixels; and
generating a first sub color demosaiced image corresponding to the first sub color and a second sub color demosaiced image corresponding to the second sub color by performing a sub interpolation to generate interpolated first sub color pixel values and interpolated second sub color pixel values for the sub target pixels and for main target pixels corresponding to the main color pixel values.

### Claim 2 (depends on 1)

The demosaicing method of claim 1, wherein performing the edge enhancement includes, for each sub target pixel:
generating a vertical filter value by filtering the input mosaic image using a vertical edge enhancement filter;
generating a horizontal filter value by filtering the input mosaic image using a horizontal edge enhancement filter;
determining an enhancement edge direction by comparing the vertical filter value and the horizontal filter value;
generating a reference chrominance value based on the input mosaic image; and
determining whether to replace the main edge direction with the enhancement edge direction.

### Claim 3 (depends on 2)

The demosaicing method of claim 2, wherein the determining whether to replace the main edge direction with the enhancement edge direction comprises:
determining to replace the main edge direction with the enhancement edge direction based on the reference chrominance value being smaller than a threshold chrominance value, the main edge direction being a horizontal direction or a vertical direction, and a difference between the vertical filter value and the horizontal filter value being greater than a first threshold value.

### Claim 4 (depends on 3)

The demosaicing method of claim 3, wherein the determining whether to replace the main edge direction with the enhancement edge direction further comprising:
determining to replace the main edge direction with the enhancement edge direction based on the reference chrominance value being smaller than the threshold chrominance value, the main edge direction being a non-direction, and the difference between the vertical filter value and the horizontal filter value being greater than a second threshold value.

### Claim 5 (depends on 2)

The demosaicing method of claim 2, wherein the determining whether to replace the main edge direction with the enhancement edge direction comprises:
determining to maintain the main edge direction without replacement with the high-frequency edge direction based on the reference chrominance value being greater than a threshold chrominance value.

### Claim 6 (depends on 2)

The demosaicing method of claim 2, wherein the generating the reference chrominance value comprises:
generating a horizontal chrominance value based on the input mosaic image;
generating a vertical chrominance value based on the input mosaic image;
determining a maximum chrominance value from among the horizontal chrominance value and the vertical chrominance value; and
determining, as the reference chrominance value, one of the horizontal chrominance value, the vertical chrominance value and the maximum chrominance value based on the vertical filter value and the horizontal filter value.

### Claim 7 (depends on 1)

The demosaicing method of claim 1, wherein the performing the main interpolation comprises:
for each sub target pixel, generating an upper coefficient value, a lower coefficient value, a left coefficient value and a right coefficient value based on a gradient of the input mosaic image;
generating an upper color difference value, a lower color difference value, a left color difference value and a right color difference value based on the upper coefficient value, the lower coefficient value, the left coefficient value, and the right coefficient value; and
generating an interpolated main color pixel value for each sub target pixel based on color difference values corresponding to the main edge direction from among the upper color difference value, the lower color difference value, the left color difference value, and the right color difference value.

### Claim 8 (depends on 7)

The demosaicing method of claim 7, wherein the generating the interpolated main color pixel value comprises:
based on the main edge direction being a vertical direction, generating the interpolated main color pixel value based on the upper color difference value and the lower color difference value;
based on the main edge direction being a horizontal direction, generating the interpolated main color pixel value based on the left color difference value and the right color difference value; and
based on the main edge direction being a non-direction, generating the interpolated main color pixel value based on the upper color difference value, the lower color difference value the left color difference value and the right color difference value.

### Claim 9 (depends on 7)

The demosaicing method of claim 7, wherein each coefficient value of the upper coefficient value, the lower coefficient value, the left coefficient value and the right coefficient value comprises:
a first coefficient value corresponding to a gradient of the main color pixel values of neighboring pixels adjacent to each sub target pixel; and
a second coefficient value corresponding to a product of a gradient of the first sub color pixel values or the second sub color pixel values of the neighboring pixels and a Laplacian of pixel values of each sub target pixel and the neighboring pixels.

### Claim 10 (depends on 7)

The demosaicing method of claim 7, wherein the generating the interpolated main color pixel value comprises:
generating direction weight values based on gradients of pixel values of neighboring pixels adjacent to each sub target pixel;
generating a weighted average value of color difference values corresponding to the main edge direction based on the direction weight values; and
generating the interpolated main color pixel value based on the weighted average value and the main color pixel values of the neighboring pixels.

### Claim 11 (depends on 1)

The demosaicing method of claim 1, wherein the performing the sub interpolation comprises:
for each sub target pixel, determining a first sub edge direction based on the main color pixel values; and
performing a first sub interpolation based on the first sub edge direction to generate the interpolated first sub color pixel values for first sub target pixels which correspond to the second sub color pixel values, and to generate the interpolated second sub color pixel values for second sub target pixels which correspond to the first sub color pixel values generating a first sub color intermediate image and a second sub color intermediate image based on the interpolated first sub color pixel values and the interpolated second sub color pixel values.

### Claim 12 (depends on 11)

The demosaicing method of claim 11, wherein the determining the first sub edge direction comprises:
for each sub target pixel, generating a vertical gradient, a horizontal gradient, a first diagonal gradient, and a second diagonal gradient based on the main color pixel values of the main color demosaiced image; and
determining the first sub edge direction as one of a vertical direction, a horizontal direction, a first diagonal direction, a second diagonal direction and a non-direction, based on a result of a comparison between the vertical gradient, the horizontal gradient, the first diagonal gradient and the second diagonal gradient.

### Claim 13 (depends on 11)

The demosaicing method of claim 11, wherein the performing the first sub interpolation comprises:
based on the first sub edge direction being a vertical direction or a horizontal direction, generating an upper color difference value, a lower color difference value, a left color difference value and a right color difference value based on the main color pixel values, and the first sub color pixel values or the second sub color pixel values;
based on the first sub edge direction being the horizontal direction, generating an interpolated sub color pixel value corresponding to each sub target pixel based on the upper color difference value and the lower color difference value, wherein the interpolated sub color pixel value comprises one of an interpolated first sub color pixel value or an interpolated second sub color pixel value; and
based on the first sub edge direction being the vertical direction, generating the interpolated sub color pixel value based on the left color difference value and the right color difference value.

### Claim 14 (depends on 13)

The demosaicing method of claim 13, wherein the generating the interpolated sub color pixel value comprises:
generating direction weight values based on gradients of the main color pixel values of neighboring pixels adjacent to each sub target pixel;
generating a weighted average value of color difference values corresponding to the horizontal direction or the vertical direction based on the direction weight values; and
generating the interpolated sub color pixel value based on the weighted average value and an interpolated main color pixel value of each sub target pixel.

### Claim 15 (depends on 11)

The demosaicing method of claim 11, wherein the performing the first sub interpolation comprises:
based on the first sub edge direction being a first diagonal direction, generating an interpolated sub color pixel value corresponding to each sub target pixel based on an arithmetic average value of color difference values of pixels adjacent to each sub target pixel in the first diagonal direction, wherein the interpolated sub color pixel value comprises one of an interpolated first sub color pixel value or an interpolated second sub color pixel value;
based on the first sub edge direction being a second diagonal direction, generating the interpolated sub color pixel value based on an arithmetic average value of color difference values of pixels adjacent to each sub target pixel in the second diagonal direction; and
based on the first sub edge direction being a non-direction, generating the interpolated sub color pixel value based on an arithmetic average value of color difference values of the pixels adjacent to each sub target pixel in the first diagonal direction and the pixels adjacent to each sub target pixel in the second diagonal direction.

### Claim 16 (depends on 11)

The demosaicing method of claim 11, wherein the performing the sub interpolation comprises:
for each main target pixel of the main target pixels, estimating a second sub edge direction based on the first sub color intermediate image or the second sub color intermediate image;
determining the second sub edge direction by performing edge enhancement based on the input mosaic image; and
generating the first sub color demosaiced image and the second sub color demosaiced image by performing a second sub interpolation based on the second sub edge direction to generate the interpolated first sub color pixel values and the interpolated second sub color pixel values for the main target pixels.

### Claim 17 (depends on 1)

The demosaicing method of claim 1, wherein a sampling rate corresponding to the main color is greater than a sampling rate corresponding the first sub color and a sampling rate corresponding to the second sub color.

### Claim 18 (depends on 1)

The demosaicing method of claim 1, wherein the main color corresponds to a green color, the first sub color corresponds to a red color, the second sub color corresponds to a blue color, and the input mosaic image has a Bayer pattern.

### Claim 19

A demosaicing method of an input mosaic image, the demosaicing method comprising:
obtaining an input image in which green pixel values corresponding to a green color, red pixel values corresponding to a red color, and blue pixel values corresponding to a blue color are arranged in a Bayer pattern;
for each sub target pixel from among sub target pixels corresponding to the red pixel values and the blue pixel values of the input mosaic image, estimating a main edge direction based on the green pixel values;
determining the main edge direction by performing edge enhancement based on the input mosaic image;
generating a green demosaiced image corresponding to the green color by performing a main interpolation based on the main edge direction to generate interpolated green color pixel values for the sub target pixels;
for each sub target pixel, determining a first sub edge direction based on the green pixel values;
generating a red intermediate image and a blue intermediate image by performing a first sub interpolation based on the first sub edge direction to generate interpolated red pixel values for the sub target pixels corresponding to the blue pixel values, and to generate interpolated blue pixel values for the sub target pixels corresponding to the red pixel values;
for each main target pixel from among main target pixels corresponding to the green pixel values, estimating a second sub edge direction based on the red intermediate image or the blue intermediate image;
determining the second sub edge direction by performing the edge enhancement based on the input mosaic image; and
generating red demosaiced image and a blue demosaiced image by performing a second sub interpolation based on the second sub edge direction to generate interpolated red pixel values for the main target pixels and interpolated blue pixel values for the main target pixels.

### Claim 20

A demosaicing device for an input mosaic image in which main color pixel values corresponding to a main color, first sub color pixel values corresponding to a first sub color, and second sub color pixel values corresponding to a second sub color are arranged in a regular pattern, the demosaicing device comprising:
a main edge estimator configured to, for each sub target pixel from among sub target pixels corresponding to the first sub color pixel values and the second sub color pixel values, estimate a main edge direction based on the main color pixel values;
a main edge enhancer configured to determine the main edge direction by performing edge enhancement based on the input mosaic image;
a main interpolator configured to generate a main color demosaiced image corresponding to the main color by performing a main interpolation based on the main edge direction to generate interpolated main color pixel values for the sub target pixels; and
a sub interpolator configured to generate a first sub color demosaiced image corresponding to the first sub color and a second sub color demosaiced image corresponding to the second sub color by performing a sub interpolation to generate interpolated first sub color pixel values and interpolated second sub color pixel values for the sub target pixels and for main target pixels corresponding to the main color pixel values.
