# US 2019/0318451 A1 · Image demosaicer and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0318451 A1
- Document ID: US20190318451A1
- Status: Published application
- Kind: A1
- Application number: 15/950,845
- Filed: 2018-04-11
- Published: 2019-10-17
- Earliest claimed: 2018-04-11
- Assignees: OmniVision Technologies, Inc.
- Applicants: OmniVision Technologies, Inc.
- Inventors: Sarvesh Swami; Yubo Duan; Chengming Liu
- Web page: https://patentgrep.com/patent/US20190318451A1

## CPC

- G06T 3/4015: Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 7/90: Determination of colour characteristics
- G06T 2207/20216: Image averaging
- G06T 2207/20224: Image subtraction
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method for demosaicing a raw image includes: (1) horizontally-interpolating a green channel formed of primary pixel-values Bg(x,y)g to yield a horizontally-interpolated green channel that includes both Bg(x,y)g and non-primary pixel-values Igh(x,y)r,b; (2) modifying each Igh(x,y)r,b, by horizontally-neighboring pixel-values, to yield a refined horizontally-interpolated green channel; (3) vertically-interpolating the green channel to yield a vertically-interpolated green channel that includes pixel-values Igv(x,y)r,b; (4) modifying each Igv(x,y)r,b by vertically-neighboring pixel-values, to yield a refined vertically-interpolated green channel; (5) generating a full-resolution green channel from the refined interpolated green channels and gradients thereof; (6) generating a full-resolution red channel by determining red pixel-values from a local-red mean value of neighboring pixel-values and the full-resolution green channel; (7) generating a full-resolution blue channel by determining pixel-values from a local-blue mean value of neighboring pixel-values and the full-resolution green channel; and (8) combining the full-resolution channels to yield a demosaiced raw image.

## Claims

### Claim 1

A method for demosaicing a raw image captured by an image sensor that includes a pixel array and a Bayer-pattern color-filter array (CFA) having a plurality of red color filters CFr, green color filters CFg, and blue color filters CFb each covering a respective one of a plurality of pixel sub-arrays of the pixel array, the pixel array including a plurality of pixels p(x,y)α located at a respective one of a plurality of coordinates (x,y)α, and beneath a color filter CFα, and each generating a respective one of a plurality of primary pixel-values Bα(x,y)α, where a is one of r, g, and b, the method comprising:
horizontally-interpolating a green channel, of the raw image, formed of primary pixel-values Bg(x,y)g of the plurality of primary pixel-values Bα(x,y)α, to yield a horizontally-interpolated green channel that includes both primary pixel-values Bg(x,y)g and non-primary pixel-values Igh(x,y)r,b;
modifying each non-primary pixel-value Igh(x,y)r,b by a plurality of horizontally-neighboring non-primary pixel-values Igh(x,y)r,b, to yield a refined horizontally-interpolated green channel;
vertically-interpolating the green channel to yield a vertically-interpolated green channel that includes both primary pixel-values Bg(x,y)g and non-primary pixel-values Igv(x Y)r,b;
modifying each non-primary pixel-value Igv(x,y)r,b by a plurality of vertically-neighboring non-primary pixel-values Igv(x,y)r,b, to yield a refined vertically-interpolated green channel;
generating a full-resolution green channel from the refined horizontally-interpolated green channel, refined vertically-interpolated green channel, a horizontal-gradient of the horizontally-interpolated green channel, and a vertical-gradient of the vertically-interpolated green channel;
generating a full-resolution red channel by determining a plurality of non-primary red pixel-values Ir(xt, yt)g,b, each corresponding to a respective one of the plurality of coordinates (xt, yt)g,b, from a local-red mean value of neighboring red primary pixel-values Br(x,y)r and the full-resolution green channel;
generating a full-resolution blue channel by determining a plurality of non-primary blue pixel-values Ib(xt,yt)g,r, each corresponding to a respective one of the plurality of coordinates (xt,yt)g,r, from a local-blue mean value of neighboring blue primary pixel-values Bb(x,y)b and the full-resolution green channel; and
combining the full-resolution red channel, the full-resolution green channel, and the full-resolution blue channel to yield a demosaiced raw image.

### Claim 2 (depends on 1)

The method of claim 1, both the horizontally-interpolating the green channel and the vertically-interpolating the green channel comprising linearly interpolating a logarithm of primary pixel-values Bg(x,y)g.

### Claim 3 (depends on 1)

The method of claim 1, each of the plurality of pixel sub-arrays having dimensions m×n, the step of modifying each non-primary pixel-value Igh(x,y)r,b comprising:
for each pixel p(x,y) beneath a red color filter of a target pixel sub-array, of the plurality of pixel sub-arrays, located between (i) a pair of horizontally-neighboring source pixel sub-arrays each beneath respective red color filters, of the CFA and horizontally collinear with the target pixel sub-array,
determining a red horizontal-supplement Hr(x,y) as a linear combination of Br(x,y)r, primary pixel-values Br(x, y1)r of horizontally-adjacent pixels p(x, y1) beneath the target pixel sub-array, and primary pixel-values Br(x, y2)r of horizontally-collinear pixels p(x, y2) beneath red color filters of the pair of horizontally-neighboring source pixel sub-arrays, where |y−y1|<n and |y−y2|≥n.

### Claim 4 (depends on 1)

The method of claim 1, each of the plurality of pixel sub-arrays having dimensions m×n, the step of modifying each non-primary pixel-value Igv (x,y)r,b, comprising:
for each pixel p(x,y) beneath a red color filter of a target pixel sub-array, of the plurality of pixel sub-arrays, located between a pair of vertically-neighboring source pixel sub-arrays each beneath a respective red color filter of the CFA and vertically collinear with the target pixel sub-array,
determining a red vertical-supplement Vr(x,y) as a linear combination of Br(x,y)r, primary pixel-values Br(x1, y)r of vertically-adjacent pixels p(x1, y) beneath the target pixel sub-array, and primary pixel-values Br(x2, y)r of vertically-collinear pixels p(x2, y) beneath red color filters of the pair of vertically-neighboring source pixel sub-arrays, where |x−x1|<m and |x−x2|≥m.

### Claim 5 (depends on 1)

The method of claim 1, further comprising determining the horizontal-gradient and the vertical-gradient by:
subtracting the raw image from the refined horizontally-interpolated green channel, and
subtracting the raw image from the refined vertical-interpolated green channel.

### Claim 6 (depends on 1)

The method of claim 1, at least one of (i) generating the full-resolution red channel and (ii) generating the full-resolution blue channel including:
binning the full-resolution green channel, and
binning the raw image.

### Claim 7 (depends on 6)

The method of claim 6, the at least one of (i) generating the full-resolution red channel and (ii) generating the full-resolution blue channel including: computing a variance of the binned full-resolution green channel and a covariance of (a) the binned full-resolution green channel and (b) the binned raw image.

### Claim 8 (depends on 6)

The method of claim 6,
binning of the full-resolution green channel yielding a binned green channel that includes a plurality of binned green pixel-values,
binning the raw image yielding a binned raw image that includes a plurality of binned red pixel-values based on the plurality of primary pixel-values Br and a plurality of binned blue pixel-values based on the plurality of primary pixel-values Bb, the local-red mean value being an average of binned red pixel-values, the local-blue mean value being an average of binned blue pixel-values.

### Claim 9 (depends on 8)

The method of claim 8,
each of the plurality of non-primary red pixel-values Ir(xt, yt)g,b having a magnitude that increases in response to an increasing covariance of the average of binned red pixel-values and an average of binned green pixel-values; and
each of the plurality of non-primary blue pixel-values Ib (xt, yt)g,r having a magnitude that increases in response to an increasing covariance of the average of binned blue pixel-values and the average of binned green pixel-values.

### Claim 10 (depends on 8)

The method of claim 8,
each of the plurality of non-primary red pixel-values Ir(xt, yt)g,b having a magnitude that decreases in response to an increasing variance of the average of binned green pixel-values; and
each of the plurality of non-primary blue pixel-values Ib(xt,yt)g,r having a magnitude that decreases in response to an increasing variance of the average of binned green pixel-values.

### Claim 11

An image demosaicer for demosaicing a raw image captured by an image sensor that includes a pixel array and a Bayer-pattern color-filter array (CFA) having a plurality of red color filters CFr, green color filters CFg, and blue color filters CFb each covering a respective one of a plurality of m×n pixel sub-arrays of the pixel array, the pixel array including a plurality of pixels p(x,y)α located at a respective one of a plurality of coordinates (x,y)α, and beneath a color filter CFα, and each generating a respective one of a plurality of primary pixel-values Bα(x,y)α, where a is one of r, g, and b:
a memory storing non-transitory computer-readable instructions and adapted to store the raw image;
a microprocessor adapted to execute the instructions to:
horizontally-interpolate a green channel, of the raw image, formed of primary pixel-values Bg(x,y)g of the plurality of primary pixel-values Bα(x,y)α, to yield a horizontally-interpolated green channel that includes both primary pixel-values Bg(x,y)g and non-primary pixel-values Igh(x,y)r,b;
modify each non-primary pixel-value Igh(x,y)r,b by a plurality of horizontally-neighboring non-primary pixel-values Igh(x,y)r,b, to yield a refined horizontally-interpolated green channel;
vertically-interpolate the green channel to yield a vertically-interpolated green channel that includes both primary pixel-values Bg(x,y)g and non-primary pixel-values Ig(x,y)r,b;
modify each non-primary pixel-value Igv(x,y)r,b by a plurality of vertically-neighboring non-primary pixel-values Igv(x,y)r,b, to yield a refined vertically-interpolated green channel;
generate a full-resolution green channel from the refined horizontally-interpolated green channel, refined vertically-interpolated green channel, a horizontal-gradient of the horizontally-interpolated green channel, and a vertical-gradient of the vertically-interpolated green channel;
generate a full-resolution red channel by determining a plurality of non-primary red pixel-values Ir (xt, yt)g,b, each corresponding to a respective one of the plurality of coordinates (xt, yt)g,b, from a local-red mean value of neighboring red primary pixel-values Br(x,y)r and the full-resolution green channel;
generate a full-resolution blue channel by determining a plurality of non-primary blue pixel-values Ib(xt, yt)g,r, each corresponding to a respective one of the plurality of coordinates (xt, yt)g,r, from a local-blue mean value of neighboring blue primary pixel-values Bb (x,y)b and the full-resolution green channel; and
combine the full-resolution red channel, the full-resolution green channel, and the full-resolution blue channel to yield a demosaiced raw image.

### Claim 12 (depends on 11)

The image demosaicer of claim 11, the microprocessor being further adapted to execute the instructions to, when both horizontally-interpolating the green channel and vertically-interpolating the green channel, linearly interpolate a logarithm of primary pixel-values Bg(x,y).

### Claim 13 (depends on 11)

The image demosaicer of claim 11, each of the plurality of pixel sub-arrays having dimensions m×n, when restoring resolution to the horizontally-interpolated green channel and for each pixel p(x,y) beneath a red color filter of a target pixel sub-array, of the plurality of pixel sub-arrays, located between (i) a pair of horizontally-neighboring source pixel sub-arrays each beneath respective red color filters, of the CFA and horizontally collinear with the target pixel sub-array, the microprocessor being further adapted to execute the instructions to:
determine a red horizontal-supplement Hr(x,y) as a linear combination of Br(x,y)r, primary pixel-values Br(x, y1)r of horizontally-adjacent pixels p(x, y1) beneath the target pixel sub-array, and primary pixel-values Br(x, y2)r of horizontally-collinear pixels p(x, y2) beneath red color filters of the pair of horizontally-neighboring source pixel sub-arrays, where |y−y1|<n and |y−y2|≥n.

### Claim 14 (depends on 11)

The image demosaicer of claim 11, each of the plurality of pixel sub-arrays having dimensions m×n, when restoring resolution to the vertically-interpolated green channel and for each pixel p(x,y) beneath a red color filter of a target pixel sub-array, of the plurality of pixel sub-arrays, located between (i) a pair of vertically-neighboring source pixel sub-arrays each beneath respective red color filters, of the CFA and vertically collinear with the target pixel sub-array, the microprocessor being further adapted to execute the instructions to:
determine a red vertical-supplement Vr(x,y) as a linear combination of Br(x,y), primary pixel-values Br(x1, y) of vertically-adjacent pixels p(x1, y) beneath the target pixel sub-array, and primary pixel-values Br(x2, y) of vertically-collinear pixels p (x2, y) beneath red color filters of the pair of vertically-neighboring source pixel sub-arrays, where |x−x1|<m and |x−x2|≥m.

### Claim 15 (depends on 11)

The image demosaicer of claim 11, the microprocessor being further adapted to execute the instructions to determine the horizontal-gradient and the vertical-gradient by:
subtracting the raw image from the refined horizontally-interpolated green channel; and
subtracting the raw image from the refined vertical-interpolated green channel.

### Claim 16 (depends on 11)

The image demosaicer of claim 11, the microprocessor being further adapted to execute the instructions to, when at least one of (i) generating the full-resolution red channel and (ii) generating the full-resolution blue channel:
bin the full-resolution green channel, and
bin the raw image.

### Claim 17 (depends on 16)

The image demosaicer of claim 16, the microprocessor being further adapted to execute the instructions to, when generating a full-resolution red channel, compute a variance of the binned full-resolution green channel and a covariance of (a) the binned full-resolution green channel and (b) the binned raw image.

### Claim 18 (depends on 16)

The image demosaicer of claim 16,
binning of the full-resolution green channel yielding a binned green channel that includes a plurality of binned green pixel-values,
binning the raw image yielding a binned raw image that includes a plurality of binned red pixel-values based on the plurality of primary pixel-values Br and a plurality of binned blue pixel-values based on the plurality of primary pixel-values Bb, the local-red mean value being an average of binned red pixel-values, the local-blue mean value being an average of binned blue pixel-values.

### Claim 19 (depends on 18)

The image demosaicer of claim 18,
each of the plurality of non-primary red pixel-values Ir(xt, yt)g,b having a magnitude that increases in response to an increasing covariance of the average of binned red pixel-values and the average of binned green pixel-values; and
each of the plurality of non-primary blue pixel-values Ib (xt, yt)g,r having a magnitude that increases in response to an increasing covariance of the average of binned blue pixel-values and the average of binned green pixel-values.

### Claim 20 (depends on 18)

The image demosaicer of claim 18,
each of the plurality of non-primary red pixel-values Ir(xt, yt)g,b having a magnitude that decreases in response to an increasing variance of the average of binned green pixel-values; and
each of the plurality of non-primary blue pixel-values Ib(xt,yt)gx having a magnitude that decreases in response to an increasing variance of the average of binned green pixel-values.