# US 2025/0159376 A1 · Hybrid image sensors with video frame interpolation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0159376 A1
- Document ID: US20250159376A1
- Status: Published application
- Kind: A1
- Application number: 18/938,080
- Filed: 2024-11-05
- Published: 2025-05-15
- Earliest claimed: 2023-11-09
- Applicants: Omnivision Technologies, Inc.
- Inventors: Tiejun Dai; Bo Mu; Menghan Guo; Wenlei Yang; Qinyi Wang; Andreas Suess; Luis Eduardo Garcia Capel
- Web page: https://patentgrep.com/patent/US20250159376A1

## CPC

- H04N 25/47: Image sensors with pixel address output; Event-driven image sensors; Selection of pixels to be read out based on image data
- H04N 23/6811: {based on the image signal}
- H04N 23/683: {performed by a processor, e.g. controlling the readout of an image memory}
- H04N 23/81: for suppressing or minimising disturbance in the image signal generation
- H04N 23/843: {Demosaicing, e.g. interpolating colour pixel values}
- H04N 25/42: by switching between different modes of operation using different resolutions or aspect ratios, e.g. switching between interlaced and non-interlaced mode
- H04N 25/46: by combining or binning pixels
- H04N 25/53: Control of the integration time
- H04N 25/616: involving a correlated sampling function, e.g. correlated double sampling [CDS] or triple sampling
- H04N 25/62: Detection or reduction of noise due to excess charges produced by the exposure, e.g. smear, blooming, ghost image, crosstalk or leakage between pixels
- H04N 25/707: Pixels for event detection
- H04N 25/76: Addressed sensors, e.g. MOS or CMOS sensors
- H04N 25/766: comprising control or output lines used for a plurality of functions, e.g. for pixel output, driving, reset or power
- H04N 25/767: Horizontal readout lines, multiplexers or registers
- H04N 25/78: Readout circuits for addressed sensors, e.g. output amplifiers or A/D converters
- H04N 25/79: Arrangements of circuitry being divided between different or multiple substrates, chips or circuit boards, e.g. stacked image sensors
- H04N 23/45: for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
- H04N 25/531: by controlling rolling shutters in CMOS SSIS
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Hybrid image sensors with video frame interpolation (and associated systems, devices, and methods) are disclosed herein. In one embodiment, an imaging system comprises one or more event vision sensor (EVS) pixels, and a plurality of CMOS image sensor (CIS) pixels. Each EVS pixel can be configured to capture event data corresponding to contrast information of light incident on the EVS pixel. Each CIS pixel can be configured to capture CIS data corresponding to intensity of light incident on the CIS pixel. The imaging system can further comprise a deblur circuit configured to deblur the CIS data captured by the plurality of CIS pixels using a first portion of the event data captured by the one or more EVS pixels, and a system processor configured to interpolate a video frame using the deblurred CIS data and all or a subset of the event data.

## Claims

### Claim 1

An imaging system, comprising:
an event driven sensing array including one or more event vision sensor (EVS) pixels, wherein each EVS pixel of the one or more EVS pixels is configured to capture event data corresponding to contrast information of light incident on the EVS pixel;
a pixel array including a plurality of CMOS image sensor (CIS) pixels arranged in one or more CIS pixel rows, wherein each CIS pixel of the plurality of CIS pixels is configured to capture CIS data corresponding to intensity of light incident on the CIS pixel;
a deblur circuit configured to deblur the CIS data captured by the plurality of CIS pixels using a first portion of the event data captured by the one or more EVS pixels; and
a system processor configured to interpolate a video frame using the deblurred CIS data and all or a subset of the event data.

### Claim 2 (depends on 1)

The imaging system of claim 1, wherein the deblur circuit is configured to (a) accumulate the event data, (b) deblur the CIS data using accumulated event data corresponding to the first portion of event data, and (b) output accumulated event data corresponding to a second portion of the event data to the system processor.

### Claim 3 (depends on 2)

The imaging system of claim 2, wherein the accumulated event data corresponding to the second portion of the event data includes an exponential of a product of (a) a running sum of the second portion of the event data and (b) a contrast threshold.

### Claim 4 (depends on 2)

The imaging system of claim 2, wherein each CIS pixel of the plurality of CIS pixels is configured to capture the CIS data during an integration period, wherein the deblur circuit is configured to accumulate the second portion of the event data during the integration period, and wherein the deblur circuit is further configured to output the accumulated event data corresponding to the second portion of the event data to the system processor at an interpolation frame timing point that occurs during the integration period.

### Claim 5 (depends on 2)

The imaging system of claim 2, wherein each CIS pixel of the plurality of CIS pixels is configured to capture the CIS data during an integration period, wherein the deblur circuit is configured to accumulate the second portion of the event data during an accumulation period preceding the integration period, and wherein the deblur circuit is further configured to output the accumulated event data corresponding to the second portion of the event data to the system processor prior to a start of the integration period.

### Claim 6 (depends on 2)

The imaging system of claim 2, wherein each CIS pixel of the plurality of CIS pixels is configured to capture the CIS data during an integration period, wherein the deblur circuit is configured to accumulate the second portion of the event data during a period of time following an end of the integration period, and wherein the deblur circuit is configured to output the accumulated event data corresponding to the second portion of the event data to the system processor at an interpolation frame timing point that occurs after an end of the integration period.

### Claim 7 (depends on 1)

The imaging system of claim 1, wherein each CIS pixel of the plurality of CIS pixels is configured to capture the CIS data during an integration period, and wherein the deblur circuit is further configured to correct the CIS data for rolling shutter distortion using a second portion of the event data accumulated by the deblur circuit prior to a start of the integration period.

### Claim 8 (depends on 7)

The imaging system of claim 7, wherein all or the subset of the event data includes the second portion of the event data, and wherein the system processor is configured to interpolate the video frame further using the rolling-shutter-distortion-corrected CIS data.

### Claim 9 (depends on 1)

The imaging system of claim 1, further comprising (i) a first frame buffer configured to store a key frame of the CIS data and (ii) a second frame buffer configured to store a frame of accumulated event data corresponding to the key frame of CIS data.

### Claim 10 (depends on 9)

The imaging system of claim 9, further comprising (i) a first switch and (ii) a trigger usable to control the first switch to selectively enable the first frame buffer to receive the key frame of the CIS data.

### Claim 11 (depends on 10)

The imaging system of claim 10, further comprising a second switch, wherein the trigger is further usable to control the second switch to selectively enable the second frame buffer to receive the frame of accumulated event data.

### Claim 12 (depends on 10)

The imaging system of claim 10, further comprising a pre-processor block configured to (a) analyze accumulated event data output from the second frame buffer to identify motion in an external scene, and (b) control the trigger to selectively enable the first switch based at least in part on identification of the motion in the external scene.

### Claim 13 (depends on 12)

The imaging system of claim 12, wherein the pre-processor block is further configured to decode the accumulated event data and/or denoise the accumulated event data.

### Claim 14 (depends on 9)

The imaging system of claim 9, further comprising a third frame buffer configured to store the interpolated video frame output from the system processor.

### Claim 15 (depends on 14)

The imaging system of claim 14, wherein the third frame buffer includes a ping-pong buffer.

### Claim 16 (depends on 1)

The imaging system of claim 1, further comprising a hybrid image sensor including the event driven sensing array and the pixel array, wherein the system processor is external to the hybrid image sensor.

### Claim 17 (depends on 16)

The imaging system of claim 16, wherein the hybrid image sensor includes the deblur circuit.

### Claim 18

A method of operating an imaging system, the method comprising:
accumulating, during an integration period for CMOS image sensor (CIS) pixels of the imaging system, first event data captured using one or more event vision sensor (EVS) pixels of the imaging system that correspond to the CIS pixels;
outputting at least a portion of the accumulated first event data to an EVS frame buffer;
deblurring, using the accumulated first event data, CIS data captured by the CIS pixels during the integration period; and
interpolating a video frame based at least in part on the deblurred CIS data and the portion of the accumulated first event data.

### Claim 19 (depends on 18)

The method of claim 18, wherein accumulating the first event data includes accumulating the first event data during a first accumulation period, and wherein the method further comprising aligning the first accumulation period with the integration period such that the first accumulation period and the integration period start at a same time.

### Claim 20 (depends on 19)

The method of claim 19, wherein aligning the first accumulation period and the integration period includes resetting the one or more EVS pixel and the CIS pixels at a same time and before (i) accumulating the first event data.

### Claim 21 (depends on 18)

The method of claim 18, wherein:
the method further comprises—
accumulating second event data captured using the one or more EVS pixels during a period of time after an end of the integration period, and
outputting, at an interpolation frame timing point occurring after the end of the integration period, at least a portion of the accumulated second event data,
wherein interpolating the video frame includes interpolating the video frame based at least in part on the portion of the accumulated second event data.

### Claim 22 (depends on 21)

The method of claim 21, further comprising resetting the one or more EVS pixels at the end of the integration period and before the period of time.

### Claim 23 (depends on 18)

The method of claim 18, wherein outputting at least the portion of the accumulated first event data to the EVS frame buffer includes outputting at least the portion of the accumulated first event data to the EVS frame buffer at an interpolation frame timing point that occurs during the integration period.

### Claim 24 (depends on 18)

The method of claim 18, wherein:
the method further comprises—
accumulating second event data captured using the one or more EVS pixels during a period of time before a start of the integration period, and
outputting at least a portion of the accumulated second event data to the EVS frame buffer at an end of the period of time and before the start of the integration period,
wherein interpolating the video frame includes interpolating the video frame based at least in part on the portion of the accumulated second event data.

### Claim 25 (depends on 24)

The method of claim 24, further comprising resetting the one or more EVS pixels and the EVS frame buffer before a start of the period of time.

### Claim 26 (depends on 18)

The method of claim 18, further comprising decoding and/or denoising at least the portion of the accumulated first event data stored to the EVS frame buffer.

### Claim 27 (depends on 18)

The method of claim 18, further comprising:
identifying motion in an external scene based on an analysis of at least the portion of the accumulated first event data stored to the EVS frame buffer; and
activating a trigger of the imaging system such that at least a portion of the CIS data captured by the CIS pixels during the integration period is stored to a CIS frame buffer of the imaging system.
