Image processing apparatus, image processing method, and image sensor

An image processing method, comprising:

• integrating a plurality of pixel values of a first exposure image for each row of a plurality of rows of the first exposure image;
• calculating a first row integration value for each row of the plurality of rows of the first exposure image based on the integration of the plurality of pixel values of the first exposure image, wherein the first exposure image is captured for a first exposure time by rolling shutter;
• integrating a plurality of pixel values of a second exposure image for each row of a plurality of rows of the second exposure image, wherein the second exposure image is captured for a second exposure time by the rolling shutter;
• calculating a second row integration value for each row of the plurality of rows of the second exposure image based on the integration of the plurality of pixel values of the second exposure image, wherein
• the second exposure image includes a subject and an angle of view common to the first exposure image, and
• the second exposure time includes a time period that overlaps with the first exposure time;
• calculating a row integration ratio based on a division of the first row integration value by the second row integration value, wherein the row integration ratio is calculated for each row of the plurality of rows of the first exposure image and for each row of the plurality of rows of the second exposure image;
• generating a reference value for each row of the plurality of rows of the first exposure image and for each row of the plurality of rows of the second exposure image;
• multiplying the row integration ratio by the reference value for each row of the plurality of rows of the first exposure image, and for each row of the plurality of rows of the second exposure image;
• integrating a plurality of multiplication results of the multiplication corresponding to the plurality of rows of the first exposure image and the plurality of rows of the second exposure image;
• calculating a sum of the plurality of multiplication results based on the integration of the plurality of multiplication results;
• estimating a flicker parameter of a flicker in the first exposure image based on the sum;
• and
• correcting the flicker in the first exposure image based on the estimated flicker parameter.

A non-transitory computer-readable medium having stored thereon computer-executable instructions that, when executed by a processor, cause the processor to execute operations, the operations comprising:

• integrating a plurality of pixel values of a first exposure image for each row of a plurality of rows of the first exposure image;
• calculating a first row integration value for each row of the plurality of rows of the first exposure image based on the integration of the plurality of pixel values of the first exposure image, wherein the first exposure image is captured for a first exposure time by rolling shutter;
• integrating a plurality of pixel values of a second exposure image for each row of a plurality of rows of the second exposure image, wherein the second exposure image is captured for a second exposure time by the rolling shutter;
• calculating a second row integration value for each row of the plurality of rows of the second exposure image based on the integration of the plurality of pixel values of the second exposure image, wherein
• the second exposure image includes a subject and an angle of view common to the first exposure image, and
• the second exposure time includes a time period that overlaps with the first exposure time;
• calculating a row integration ratio based on a division of the first row integration value by the second row integration value, wherein the row integration ratio is calculated for each row of the plurality of rows of the first exposure image and for each row of the plurality of rows of the second exposure image;
• generating a reference value for each row of the plurality of rows of the first exposure image and for each row of the plurality of rows of the second exposure image;
• multiplying the row integration ratio by the reference value for each row of the plurality of rows of the first exposure image, and for each row of the plurality of rows of the second exposure image;
• integrating a plurality of multiplication results of the multiplication corresponding to the plurality of rows of the first exposure image and the plurality of rows of the second exposure image;
• calculating a sum of the plurality of multiplication results based on the integration of the plurality of multiplication results;
• estimating a flicker parameter of a flicker in the first exposure image based on the sum;
• and
• correcting the flicker in the first exposure image based on the estimated flicker parameter.

An image sensor, comprising:

• a pixel unit that comprises a plurality of pixels;
• and
• a central processing unit (CPU), wherein
• the pixel unit is configured to:
• set an exposure condition for each pixel of the plurality of pixels;
• obtain a first exposure image based on a reading operation of a first group of pixels of the plurality of pixels, wherein the first group of pixels is set to a first exposure time;
• obtain a second exposure image based on a reading operation of a second group of pixels of the plurality of pixels, wherein the second group of pixels is set to a second exposure time;
• and
• output the first exposure image and the second exposure image to the CPU, the CPU is configured to:
• integrate a plurality of pixel values of the first exposure image for each row of a plurality of rows of the first exposure image;
• calculate a first row integration value for each row of the plurality of rows of the first exposure image based on the integration of the plurality of pixel values of the first exposure image, wherein the first exposure image is captured for the first exposure time by rolling shutter;
• integrate a plurality of pixel values of the second exposure image for each row of a plurality of rows of the second exposure image, wherein the second exposure image is captured for the second exposure time by the rolling shutter;
• calculate a second row integration value for each row of the plurality of rows of the second exposure image based on the integration of the plurality of pixel values of the second exposure image, wherein
• the second exposure image includes a subject and an angle of view common to the first exposure image, and
• the second exposure time includes a time period that overlaps with the first exposure time;
• calculate a row integration ratio based on a division of the first row integration value by the second row integration value, wherein the row integration ratio is calculated for each row of the plurality of rows of the first exposure image and for each row of the plurality of rows of the second exposure image;
• generate a reference value for each row of the plurality of rows of the first exposure image and for each row of the plurality of rows of the second exposure image;
• multiply the row integration ratio by the reference value for each row of the plurality of rows of the first exposure image, and for each row of the plurality of rows of the second exposure image;
• integrate a plurality of multiplication results of the multiplication corresponding to the plurality of rows of the first exposure image and the plurality of rows of the second exposure image;
• calculate a sum of the plurality of multiplication results based on the integration of the plurality of multiplication results;
• estimate a flicker parameter of a flicker in the first exposure image based on the sum;
• and
• correct the flicker in the first exposure image based on the estimated flicker parameter.

An image processing apparatus, comprising:

• a central processing unit (CPU) configured to:
• integrate a plurality of pixel values of a first exposure image for each row of a plurality of rows of the first exposure image;
• calculate a first row integration value for each row of the plurality of rows of the first exposure image based on the integration of the plurality of pixel values of the first exposure image, wherein the first exposure image is captured for a first exposure time by rolling shutter;
• integrate a plurality of pixel values of a second exposure image for each row of a plurality of rows of the second exposure image, wherein the second exposure image is captured for a second exposure time by the rolling shutter;
• calculate a second row integration value for each row of the plurality of rows of the second exposure image based on the integration of the plurality of pixel values of the second exposure image, wherein
• the second exposure image includes a subject and an angle of view common to the first exposure image, and
• the second exposure time includes a time period that overlaps with the first exposure time;
• calculate a row integration ratio based on a division of the first row integration value by the second row integration value, wherein the row integration ratio is calculated for each row of the plurality of rows of the first exposure image, and for each row of the plurality of rows of the second exposure image;
• generate a reference value for each row of the plurality of rows of the first exposure image and for each row of the plurality of rows of the second exposure image;
• multiply the row integration ratio by the reference value for each row of the plurality of rows of the first exposure image, and for each row of the plurality of rows of the second exposure image;
• integrate a plurality of multiplication results of the multiplication corresponding to the plurality of rows of the first exposure image and the plurality of rows of the second exposure image;
• calculate a sum of the plurality of multiplication results based on the integration of the plurality of multiplication results;
• estimate a flicker parameter of a flicker in the first exposure image based on the sum and a least-square method;
• and
• correct the flicker in the first exposure image based on the estimated flicker parameter.