# US 2003/0222995 A1 · Method and apparatus for real time identification and correction of pixel defects for image sensor arrays

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2003/0222995 A1
- Document ID: US20030222995A1
- Status: Published application
- Kind: A1
- Application number: 10/160,556
- Filed: 2002-06-04
- Published: 2003-12-04
- Earliest claimed: 2002-06-04
- Assignees: Micron Technology, Inc.
- Inventors: Michael Kaplinsky; Igor Subbotin
- Web page: https://patentgrep.com/patent/US20030222995A1

## CPC

- H04N 25/683: by defect estimation performed on the scene signal, e.g. real time or on the fly detection
- G06T 5/20: using local operators
- G06T 5/77: Retouching; Inpainting; Scratch removal
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An image processing system and method compares each pixel of an image obtained from an image sensor array with at least eight surrounding pixels of the same color in the filter array. If the signal of a given pixel is larger than the respective signals of all eight surrounding pixels of the same color, then the value of that central pixel signal is substituted with the maximum signal value among the surrounding eight pixels of the same color. Similarly, if the signal of a given pixel is smaller than the respective signals of all eight surrounding pixels of the same color, then the value of that central pixel signal is substituted with the minimum signal value among the surrounding eight pixels of the same color.

## Claims

### Claim 1

An apparatus for correcting pixel image data comprising:
a memory structure for receiving a plurality of lines of image data from an image obtained from an image sensor array, each line containing signal value data corresponding to a line of pixels in the array; and
a correction section which receives the image data from the memory structure and in which the signal value of a tested pixel in the array is compared with the signal values of at least eight surrounding pixels, and which, if the signal value of the tested pixel is larger than all of the signal values of the eight surrounding pixels, the signal value of the tested pixel is replaced with the largest signal value from among the eight surrounding pixels, and which, if the signal value of the pixel being tested is smaller than all of the signal values of the eight surrounding pixels, the signal value of the tested pixel is replaced with the smallest signal value from among the eight surrounding pixels.

### Claim 2 (depends on 1)

The apparatus according to claim 1, wherein the eight surrounding pixels are immediately adjacent to the pixel being tested in the horizontal, vertical and diagonal directions in a two dimensional area for a monochrome image.

### Claim 3 (depends on 1)

The apparatus according to claim 1, wherein the eight surrounding pixels are spaced one pixel away from the tested pixel in the horizontal, vertical and diagonal directions in a two dimensional area, and wherein the eight surrounding pixels are of the same color in a color filter array as the tested pixel.

### Claim 4 (depends on 1)

The apparatus according to claim 1, further comprising a controller for controlling the memory structure to read out the image data to the correction section so as to sequentially test the signal value of each pixel in a line of image data.

### Claim 5 (depends on 4)

The apparatus according to claim 4, wherein the controller further controls an input of image data into the memory structure such that a new line of image data is being written into the memory structure to replace an oldest line of image data as the oldest line of image data is being read out, and upon completion of testing the signal value of each pixel in a line of image data, further controls the memory structure to read out the updated image data so that the correction section sequentially tests the signal values for the pixels in a next line of image data.

### Claim 6 (depends on 1)

The apparatus of claim 1, further comprising an image processing section which performs an image processing operation on the tested pixel of the image data after the tested pixel has been tested.

### Claim 7 (depends on 16)

The apparatus of claim 16, wherein the image processing circuit performs a color processing operation.

### Claim 8 (depends on 16)

The apparatus of claim 16, wherein the image processing circuit performs a color interpolation operation.

### Claim 9 (depends on 16)

The apparatus of claim 16, wherein the image processing circuit performs a white balancing operation.

### Claim 10 (depends on 16)

The apparatus of claim 16, wherein the image processing circuit performs a sharpness filtering operation.

### Claim 11

An apparatus for correcting pixel image data, comprising:
a plurality of memory banks each for receiving and temporarily storing a line of image data of an image obtained from an image sensor array, each line containing signal value data respectively corresponding to a line of pixels in the array;
a plurality of sets of shift registers for each receiving the image data from the plurality of memory banks;
a correction circuit for testing the signal value of a pixel to be tested in the image data in one of the sets of shift registers to determine if the signal value of the tested pixel is defective, and if found to be defective, for correcting the defective signal value.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein the correction circuit tests the pixel to be tested by comparing a signal value of the pixel to be tested with the respective signal values of at least eight pixels surrounding the tested pixel and having the same color type as the tested pixel.

### Claim 13 (depends on 12)

The apparatus according to claim 12, wherein the plurality of memory banks includes three memory banks and wherein the eight surrounding pixels are immediately adjacent to the pixel being tested in the horizontal, vertical and diagonal directions in a two dimensional area for a monochrome image.

### Claim 14 (depends on 12)

The apparatus according to claim 12, wherein the plurality of memory banks includes five memory banks and wherein the eight surrounding pixels are spaced one pixel away from the tested pixel in the horizontal, vertical and diagonal directions in a two dimensional area, and wherein the eight surrounding pixels are of the same color in a color filter array as the tested pixel.

### Claim 15 (depends on 12)

The apparatus of claim 12, wherein the correction circuit redefines the signal value of the tested pixel to be equal to the maximum signal value from among the signal values of the eight surrounding pixels if the signal value of the tested pixel is larger than all of the signal values for the eight surrounding pixels.

### Claim 16 (depends on 12)

The apparatus of claim 12, wherein the correction circuit redefines the signal value of the tested pixel to be equal to the minimum signal value from among the signal values of the eight surrounding pixels if the signal value of the tested pixel is smaller than all of the signal values for the eight surrounding pixels.

### Claim 17 (depends on 11)

The apparatus according to claim 11, further comprising:
a delay register for inputting a line of image data directly from the image sensor array into one of the plurality of sets of shift registers in column synchrony with the image data being inputted into the shift registers from the memory banks.

### Claim 18 (depends on 11)

The apparatus in accordance with claim 11, further comprising:
an input multiplexer for managing the writing of image data into each of the memory banks;
an output multiplexer for managing the reading out of the image data from the memory banks into the shift registers;
an address generator for controlling the addresses of the data being written into and read out of the memory banks.

### Claim 19 (depends on 11)

The apparatus according to claim 11, further comprising a controller for controlling the output of the image data from the plurality of memory banks so that the testing and correcting, if necessary, are performed sequentially for the signal values for each pixel in a line of image data.

### Claim 20 (depends on 19)

The apparatus according to claim 19, wherein the controller further controls an input of image data into the memory banks to write a new line of image data into the memory bank containing an oldest line of image data so as to overwrite the oldest line of image data as it is being outputted from the relevant memory bank, and upon completion of testing the signal value of each pixel in a line of image data, further controls the output of the updated image data from the plurality of memory bank so that the testing and correcting, if necessary, are performed sequentially for the signal values for each pixel in a next line of image data.

### Claim 21 (depends on 11)

The apparatus of claim 11, further comprising
an image processing circuit for performing an image processing operation on a pixel of the image data already tested, the image processing circuit using a portion of the shift registers which is offset from the shift registers in which the tested pixel has been tested.

### Claim 22 (depends on 21)

The apparatus of claim 21, wherein the image processing circuit performs a color processing operation.

### Claim 23 (depends on 21)

The apparatus of claim 21, wherein the image processing circuit performs a color interpolation operation.

### Claim 24 (depends on 21)

The apparatus of claim 21, wherein the image processing circuit performs a white balancing operation.

### Claim 25 (depends on 21)

The apparatus of claim 21, wherein the image processing circuit performs a sharpness filtering operation.

### Claim 26

A method for correcting pixel image data, comprising:
a) obtaining at least three lines of image data containing signal values for a corresponding number of lines of an image obtained from a pixel array in an image sensor;
b) comparing a signal value of a pixel to be tested from among the at least three lines of signal values with the respective signal values of at least eight pixels from among the at least three lines of signal values and which surround the tested pixel, having the eight surrounding pixels the same color type as the tested pixel;
c) if the signal value of the tested pixel is larger than all of the signal values for the eight surrounding pixels, redefining the signal value of the tested pixel to be equal to the maximum signal value from among the signal values of the eight surrounding pixels;
d) if the signal value of the tested pixel is smaller than all of the signal values for the eight surrounding pixels, redefining the signal value of the tested pixel to be equal to the minimum signal value from among the signal values of the eight surrounding pixels.

### Claim 27 (depends on 26)

The method according to claim 26, wherein the eight surrounding pixels are immediately adjacent to the tested pixel in the horizontal, vertical and diagonal directions in a two dimensional area for a monochrome image.

### Claim 28 (depends on 26)

The method according to claim 26, wherein the eight surrounding pixels are spaced one pixel away from the pixel being tested in the horizontal, vertical and diagonal directions in a two dimensional area, and wherein the eight surrounding pixels are of the same color in a color filter array as the tested pixel.

### Claim 29 (depends on 26)

The method according to claim 26, wherein the comparing act is not performed for the signal values of the pixels from the outermost two rows and columns in the image sensor pixel array.

### Claim 30 (depends on 30)

The method according to claim 30, further comprising
e) repeating acts a) through d) for a next pixel in the same line of image data as the tested pixel, until all the pixels in the line have been tested.

### Claim 31 (depends on 30)

The method according to claim 30, further comprising:
f) obtaining a new line of signal values for an new line of image data from the image sensor pixel array; and
g) repeating act e) on a next line of image data and also repeating act f) until all the signal values for each pixel in each line of image data has been tested.

### Claim 32 (depends on 31)

The method according to claim 31, further comprising
verifying that no cluster of defective pixels is present after testing each pixel in each line of image data by
h) selecting a first pixel from among the tested image data;
i) selecting a second pixel from among the closest surrounding pixels to the first pixel and which is of the same color type as the first pixel;
j) comparing the difference in signal values between the first pixel and second pixel;
k) comparing the absolute value of the difference to a threshold value, wherein a cluster of defective pixels is determined to be present if the absolute value of the difference is greater than the threshold value; and
l) repeating act h) through k) for each pixel in the tested image data.

### Claim 33

A method for correcting pixel image data, comprising:
writing at least three lines of image data containing signal values for a corresponding number of lines of an image obtained from a pixel array in an image sensor, into a plurality of memory banks;
sequentially inputting the image data from the memory banks into a plurality of sets of shift registers;
comparing a signal value of a pixel to be tested in the shift registers with the signal values of at least eight surrounding pixels in the shift registers and which are of a same color filter type as the tested pixel;
if the signal value of the pixel being tested is larger than all of the signal values of the eight surrounding pixels, correcting the signal value of the pixel being tested to be equal to the largest signal value from among the eight surrounding pixels; and
if the signal value of the pixel being tested is smaller than all of the signal values of the eight surrounding pixels, correcting the signal value of the pixel being tested to be equal to the smallest signal value from among the eight surrounding pixels.

### Claim 34 (depends on 33)

The method according to claim 33, further comprising inputting a line of image data directly into a set of shift registers separately from the plurality of memory banks, but in column synchrony with the lines of image data being inputted into the shift registers from the memory banks, wherein a portion the signal values of the at least eight surrounding pixels for being compared with the signal value of the tested pixel are selected from among the image data inputted directly into the shift registers.

### Claim 35 (depends on 33)

The method according to claim 33, wherein the eight surrounding pixels are immediately adjacent to the pixel being tested in the horizontal, vertical and diagonal directions in a two dimensional area for a monochrome image.

### Claim 36 (depends on 33)

The method according to claim 33, wherein the eight surrounding pixels are spaced one pixel away from the pixel being tested in the horizontal, vertical and diagonal directions in a two dimensional area, and wherein the eight surrounding pixels are of the same color in a color filter array as the pixel being tested.

### Claim 37 (depends on 33)

The method according to claim 33, further comprising, upon correcting the signal value of the tested pixel, replacing the corrected value in the corresponding location in the corresponding memory bank.

### Claim 38 (depends on 33)

The method according to claim 33, further comprising, upon correcting the signal value of the tested pixel, replacing the corrected value in the corresponding location in the shift registers.

### Claim 39 (depends on 33)

The method according to claim 33, further comprising
controlling an input of lines of image data into the memory banks to write a new line of image data into a memory bank containing an oldest line of data so as to overwrite the oldest line of image data as it is being outputted from the relevant memory bank; and
upon completion of testing the signal value of each pixel in a line of image data, controlling the output of the updated image data from the plurality of memory banks so that the comparing act and the correcting act, if necessary, are performed sequentially for the signal values for each pixel in a next line of image data.

### Claim 40 (depends on 33)

The method according to claim 33, further comprising controlling the output of the image data from the memory banks so that the comparing act and the correction act, if necessary, are performed sequentially for the signal values for each pixel in a line of image data.

### Claim 41 (depends on 40)

The method according to claim 40, further comprising
replacing the oldest line of image data in the relevant memory bank with a new line of image data, and
repeating the controlling act on a next line of image data and also repeating the replacing act until all the signal values for each pixel in each line of image data in the image has been tested.

### Claim 42 (depends on 33)

The method according to claim 33, further comprising, after the comparing act and, if necessary, the correcting act have been performed for the tested pixel, performing a subsequent image processing function using an offset portion of the shift registers from those used to test the tested pixel.

### Claim 43 (depends on 42)

The method according to claim 42, wherein the subsequent image processing function is a color processing function.

### Claim 44 (depends on 42)

The method according to claim 42, wherein the subsequent image processing function is a color interpolation function.

### Claim 45 (depends on 42)

The method according to claim 42, wherein the subsequent image processing function is a sharpness filtering function.

### Claim 46 (depends on 42)

The method according to claim 42, wherein the subsequent image processing function is a white balancing function

### Claim 47

An apparatus for correcting pixel image data comprising:
a plurality of memory banks each for receiving and temporarily storing a line of image data of an image obtained from an image sensor array, each line containing signal value data corresponding to a line of pixels in the array; and
a correction section which receives the image data from the plurality of memory banks, compares the signal value of a tested pixel in the array with the signal values of at least eight surrounding pixels, and replaces the signal value of the tested pixel with the largest signal value from among the eight surrounding pixels if the value of the tested pixel is larger than all of the signal values of the eight surrounding pixels.

### Claim 48 (depends on 47)

The apparatus according to claim 47, comprising three memory banks and wherein the eight surrounding pixels are immediately adjacent to the pixel being tested in the horizontal, vertical and diagonal directions in a two dimensional area for a monochrome image.

### Claim 49 (depends on 47)

The apparatus according to claim 47, comprising five memory banks and wherein the eight surrounding pixels are spaced one pixel away from the tested pixel in the horizontal, vertical and diagonal directions in a two dimensional area, and wherein the eight surrounding pixels are of the same color in a color filter array as the tested pixel.

### Claim 50

An apparatus for correcting pixel image data comprising:
a plurality of memory banks each for receiving and temporarily storing a line of image data of an image obtained from an image sensor array, each line containing signal value data corresponding to a line of pixels in the array; and
a correction section which receives the image data from the plurality of memory banks, compares the signal value of a tested pixel in the array with the signal values of at least eight surrounding pixels, and replaces the signal value of the tested pixel with the smallest signal value from among the eight surrounding pixels if the value of the tested pixel is smaller than all of the signal values of the eight surrounding pixels.

### Claim 51 (depends on 50)

The apparatus according to claim 50, comprising three memory banks and wherein the eight surrounding pixels are immediately adjacent to the pixel being tested in the horizontal, vertical and diagonal directions in a two dimensional area for a monochrome image.

### Claim 52 (depends on 50)

The apparatus according to claim 50, comprising five memory banks and wherein the eight surrounding pixels are spaced one pixel away from the tested pixel in the horizontal, vertical and diagonal directions in a two dimensional area, and wherein the eight surrounding pixels are of the same color in a color filter array as the tested pixel.

### Claim 53

A method for correcting pixel image data comprising:
a) obtaining at least three lines of image data containing signal values for a corresponding number of lines of image data of an image obtained from a pixel array in an image sensor;
b) comparing a signal value of a pixel to be tested from among the at least three lines of image data with the respective signal values of at least eight pixels from among the at least three lines of image data and which surround the tested pixel, the eight surrounding pixels having the same color type as the tested pixel; and
c) replacing the signal value of the tested pixel with the maximum signal value from among the signal values of the eight surrounding pixels if the signal value of the tested pixel is larger than all of the signal values for the eight surrounding pixels.

### Claim 54 (depends on 53)

The method according to claim 53, wherein the eight surrounding pixels are immediately adjacent to the tested pixel in the horizontal, vertical and diagonal directions in a two dimensional area for a monochrome image.

### Claim 55 (depends on 53)

The method according to claim 53, wherein the eight surrounding pixels are spaced one pixel away from the pixel being tested in the horizontal, vertical and diagonal directions in a two dimensional area, and wherein the eight surrounding pixels are of the same color in a color filter array as the tested pixel.

### Claim 56

A method for correcting image data due to defective pixels in an image sensor array, comprising:
a) obtaining at least three lines of image data containing signal values for a corresponding number of lines of image data of an image obtained from a pixel array in an image sensor;
b) comparing a signal value of a pixel to be tested from among the at least three lines of image data with the respective signal values of at least eight pixels from among the at least three lines of image data and which surround the tested pixel, the eight surrounding pixels having the same color type as the tested pixel; and
c) replacing the signal value of the tested pixel with the minimum signal value from among the signal values of the eight surrounding pixels if the signal value of the tested pixel is smaller than all of the signal values for the eight surrounding pixels.

### Claim 57 (depends on 56)

The method according to claim 56, wherein the eight surrounding pixels are immediately adjacent to the tested pixel in the horizontal, vertical and diagonal directions in a two dimensional area for a monochrome image.

### Claim 58 (depends on 56)

The method according to claim 56, wherein the eight surrounding pixels are spaced one pixel away from the pixel being tested in the horizontal, vertical and diagonal directions in a two dimensional area, and wherein the eight surrounding pixels are of the same color in a color filter array as the tested pixel.

### Claim 59

An apparatus for correcting pixel image data, comprising:
a plurality of memory banks each for receiving and temporarily storing a line of image data from an image sensor array, each line containing signal values corresponding to a line of pixels in the array; and
a correction circuit which identifies and corrects signal values produced by defective pixels in the array without using a predetermined and stored map of defective pixel locations and without use of threshold values.

### Claim 60

An image processing apparatus comprising:
an image sensing unit for obtaining an image and outputting an image signal which includes pixel image data containing signal values for each line of the image;
an image processor for processing the image signal; and
a controller for controlling the image sensing unit and the image processor, and
wherein the image processor includes a defective pixel data correction circuit which receives the pixel image data and compares the signal value of a tested pixel with the signal values of at least eight pixels surrounding the tested pixel in the image, and which, if the signal value of the tested pixel is larger than all of the signal values of the eight surrounding pixels, replaces the signal value of the tested pixel with the largest signal value from among the eight surrounding pixels, and which, if the signal value of the tested pixel is smaller than all of the signal values of the eight surrounding pixels, replaces the signal value of the tested pixel with the smallest signal value from among the eight surrounding pixels.

### Claim 61

A processing system, comprising:
a processor for receiving an processing image data; and
an image data generator for supplying image data to the processor, the image data generator comprising
an image sensing unit for obtaining an image and outputting an image signal,
an image processor for processing the image signal, and
a controller for controlling the image sensing unit and the image processor,
wherein the image processor includes a defective pixel data correction circuit which receives the pixel image data and compares the signal value of a tested pixel with the signal values of at least eight pixels surrounding the tested pixel in the image, and which, if the signal value of the tested pixel is larger than all of the signal values of the eight surrounding pixels, replaces the signal value of the tested pixel with the largest signal value from among the eight surrounding pixels, and which, if the signal value of the tested pixel is smaller than all of the signal values of the eight surrounding pixels, replaces the signal value of the tested pixel with the smallest signal value from among the eight surrounding pixels.
