Hybrid image sensors with on-chip image deblur and rolling shutter distortion correction

An image sensor, comprising:

• an event driven sensing array including one or more event vision sensor (EVS) pixels, wherein each EVS pixel of the one or more EVS pixels is configured to capture event data corresponding to contrast information of light incident on the EVS pixel;
• a pixel array including a plurality of CMOS image sensor (CIS) pixels, wherein each CIS pixel of the plurality of CIS pixels is configured to capture CIS data corresponding to intensity of light incident on the CIS pixel;
• and
• a rolling shutter distortion correction circuit including—
• a counter usable, for each EVS pixel of the one or more EVS pixels, to compute a running sum of events detected by the EVS pixel during a period of time extending from a start of an image frame to a start of an exposure period for one or more CIS pixels of the plurality of CIS pixels that correspond to the EVS pixel, and
• an integration buffer configured, for each EVS pixel of the one or more EVS pixels, to store the running sum corresponding to the EVS pixel.

A method of operating an image sensor, the method comprising:

• capturing, during an exposure period corresponding to an image frame, CMOS image sensor (CIS) data using CIS pixels of one or more CIS pixel rows of a pixel array of the image sensor, wherein the CIS data corresponds to intensity of light incident on the CIS pixels over the exposure period;
• capturing, during a period of time extending between a start of the image frame and a start of the exposure period, event vision sensor (EVS) data using one or more EVS pixels of an event driven sensing array of the image sensor, wherein the EVS data includes events detected by the one or more EVS pixels during the period of time, and wherein each event represents temporal contrast of light incident on the one or more EVS pixels that exceeds a threshold;
• and
• correcting, using a rolling shutter distortion correction (RSDC) circuit of the image sensor, the CIS data for rolling shutter distortion (i) based on the EVS data and (ii) internally within the image sensor.

An image sensor, comprising:

• an event driven sensing array including one or more event vision sensor (EVS) pixels, wherein each EVS pixel of the one or more EVS pixels is configured to capture event data corresponding to contrast information of light incident on the EVS pixel;
• a pixel array including a plurality of CMOS image sensor (CIS) pixels arranged in one or more CIS pixel rows, wherein each CIS pixel of the plurality of CIS pixels is configured to capture CIS data corresponding to intensity of light incident on the CIS pixel;
• a deblur and rolling shutter distortion correction circuit configured to (i) generate deblurred image data by deblurring the CIS data captured by the plurality of CIS pixels using the event data captured by the one or more EVS pixels and (ii) generate rolling-shutter-distortion-corrected image data by correcting the CIS data captured by the plurality of CIS pixels for rolling shutter distortion using the event data captured by the one or more EVS pixels, wherein the deblur and rolling shutter distortion correction circuit includes—
• a counter usable to compute a running sum of events detected in event data captured by an EVS pixel of the one or more EVS pixels during an exposure period for one or more CIS pixels of the plurality of CIS pixels that correspond to the EVS pixel, and
• an integration buffer configured to store the running sum;
• and
• a physical interface usable to output the rolling-shutter-distortion-corrected image data from the image sensor.