# US 2025/0159371 A1 · Hybrid image sensors with on-chip image deblur

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0159371 A1
- Document ID: US20250159371A1
- Status: Published application
- Kind: A1
- Application number: 18/938,208
- Filed: 2024-11-05
- Published: 2025-05-15
- Earliest claimed: 2023-11-09
- Assignees: OmniVision Technologies, Inc.
- Applicants: Omnivision Technologies, Inc.
- Inventors: Tiejun Dai; Bo Mu; Menghan Guo; Wenlei Yang; Qinyi Wang; Luis Eduardo Garcia Capel; Rui Jiang; Andreas Suess
- Web page: https://patentgrep.com/patent/US20250159371A1

## CPC

- H04N 25/47: Image sensors with pixel address output; Event-driven image sensors; Selection of pixels to be read out based on image data
- H04N 23/6811: {based on the image signal}
- H04N 23/683: {performed by a processor, e.g. controlling the readout of an image memory}
- H04N 23/81: for suppressing or minimising disturbance in the image signal generation
- H04N 23/843: {Demosaicing, e.g. interpolating colour pixel values}
- H04N 25/42: by switching between different modes of operation using different resolutions or aspect ratios, e.g. switching between interlaced and non-interlaced mode
- H04N 25/46: by combining or binning pixels
- H04N 25/53: Control of the integration time
- H04N 25/616: involving a correlated sampling function, e.g. correlated double sampling [CDS] or triple sampling
- H04N 25/62: Detection or reduction of noise due to excess charges produced by the exposure, e.g. smear, blooming, ghost image, crosstalk or leakage between pixels
- H04N 25/707: Pixels for event detection
- H04N 25/76: Addressed sensors, e.g. MOS or CMOS sensors
- H04N 25/766: comprising control or output lines used for a plurality of functions, e.g. for pixel output, driving, reset or power
- H04N 25/767: Horizontal readout lines, multiplexers or registers
- H04N 25/78: Readout circuits for addressed sensors, e.g. output amplifiers or A/D converters
- H04N 25/79: Arrangements of circuitry being divided between different or multiple substrates, chips or circuit boards, e.g. stacked image sensors
- H04N 23/45: for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
- H04N 25/531: by controlling rolling shutters in CMOS SSIS
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Hybrid image sensors with on-chip image deblur (and associated systems, devices, and methods) are disclosed herein. In one embodiment, an image sensor includes (a) a plurality of CMOS image sensor (CIS) pixels configured to capture CIS data corresponding intensity of light incident on CIS pixels of the plurality of CIS pixels, (b) an event vision sensor (EVS) pixel configured to capture EVS data corresponding to events detected in light incident on the EVS pixel, and (c) a deblur circuit configured to generate deblurred image data based on the CIS data and an accumulation of events in the EVS data. The deblur circuit can be configured to compute the accumulation of events, such as using an event-based double integral model.

## Claims

### Claim 1

An image sensor, comprising:
an event driven sensing array including one or more event vision sensor (EVS) pixels arranged in one or more EVS pixel rows, wherein each EVS pixel of the one or more EVS pixels is configured to capture event data corresponding to contrast information of light incident on the EVS pixel;
a pixel array including a plurality of CMOS image sensor (CIS) pixels arranged in one or more CIS pixel rows, wherein each CIS pixel of the plurality of CIS pixels is configured to capture CIS data corresponding to intensity of light incident on the CIS pixel;
a deblur circuit configured to generate deblurred image data by deblurring the CIS data captured by the plurality of CIS pixels using the event data captured by the one or more EVS pixels; and
a physical interface usable to output the deblurred image data from the image sensor.

### Claim 2 (depends on 1)

The image sensor of claim 1, wherein the deblur circuit includes a counter usable to compute a running sum of events detected in event data captured by an EVS pixel of the one or more EVS pixels during an event accumulation period.

### Claim 3 (depends on 2)

The image sensor of claim 2, wherein the counter includes an integer counter.

### Claim 4 (depends on 3)

The image sensor of claim 3, wherein the integer counter includes an up/down counter.

### Claim 5 (depends on 2)

The image sensor of claim 2, wherein the counter is clocked by events in the event data captured by the EVS pixel during the accumulation period.

### Claim 6 (depends on 2)

The image sensor of claim 2, wherein the deblur circuit further includes a buffer configured to store the running sum.

### Claim 7 (depends on 2)

The image sensor of claim 2, wherein the deblur circuit further includes a product computation block configured to compute a product of the running sum by a contrast threshold parameter.

### Claim 8 (depends on 7)

The image sensor of claim 7, wherein the deblur circuit further includes an exponential computation block configured to compute an exponential of the product.

### Claim 9 (depends on 8)

The image sensor of claim 8, wherein the deblur circuit further includes an integration computation block configured to compute the integral of the exponential of the product over time.

### Claim 10 (depends on 9)

The image sensor of claim 9, wherein the integration computation block is configured to continuously compute the integral over the event accumulation period.

### Claim 11 (depends on 9)

The image sensor of claim 9, wherein the deblur circuit further includes a buffer configured to store results of the integral computed by the integration computation block.

### Claim 12 (depends on 1)

The image sensor of claim 1, wherein:
the deblurred image data includes a latent frame of the CIS data;
the deblur circuit includes a latent frame computation block configured to compute the latent frame based at least in part on (i) first event data captured by an EVS pixel of an EVS pixel row of the one of more EVS pixel rows and (ii) first CIS data captured by a CIS pixel of a CIS pixel row corresponding to the EVS pixel row;
the first CIS data is captured by the CIS pixel over an exposure period; and
the first event data is captured by the EVS pixel over an event accumulation period that is aligned with the exposure period.

### Claim 13 (depends on 1)

The image sensor of claim 1, further comprising:
first row/column control circuitry and first readout circuitry, each corresponding to the pixel array; and
second row/column control circuitry and second readout circuitry different from the first row/column control circuitry and the first readout circuitry, respectively, and each corresponding to the event driven sensing array.

### Claim 14 (depends on 13)

The image sensor of claim 13, further comprising a common control block configured to synchronize operations of the first row/column control circuitry, the first readout circuitry, the second row/column control circuitry, and the second readout circuitry with one another.

### Claim 15 (depends on 1)

The image sensor of claim 1, wherein:
the physical interface is a first physical interface; and
the image sensor further includes a second physical interface separate from the first physical interface and usable to output the event data.

### Claim 16 (depends on 1)

The image sensor of claim 1, wherein the physical interface is further usable to output the event data.

### Claim 17 (depends on 1)

The image sensor of claim 1, further comprising a multiplexer that is controllable to selectively output the deblurred image data or the CIS data from the image sensor.

### Claim 18 (depends on 1)

The image sensor of claim 1, further comprising a multiplexer that is controllable to selectively output the event data from the image sensor.

### Claim 19

An imaging system, comprising:
an image sensor including—
a plurality of CMOS image sensor (CIS) pixels configured to capture CIS data corresponding intensity of light incident on CIS pixels of the plurality of CIS pixels,
an event vision sensor (EVS) pixel configured to capture EVS data corresponding to events detected in light incident on the EVS pixel, and
a deblur circuit configured to generate deblurred image data based on the CIS data and an accumulation of events in the EVS data; and
an image signal processor configured to interface with the image sensor and receive the deblurred image data from the image sensor.

### Claim 20 (depends on 19)

The imaging system of claim 19, wherein the deblur circuit is configured to compute the accumulation of events using an event-based double integral model.

### Claim 21

A method of operating an image sensor, the method comprising:
capturing CMOS image sensor (CIS) data using CIS pixels of one or more CIS pixel rows of a pixel array of the image sensor, wherein the CIS data corresponds to intensity of light incident on the CIS pixels over an exposure period;
capturing event vision sensor (EVS) data using one or more EVS pixels of an EVS pixel row of an event driven sensing array of the image sensor, wherein the EVS data includes events detected by the one or more EVS pixels during an event accumulation period,
and wherein each event represents temporal contrast of light incident on the one or more EVS pixels that exceeds a threshold; and
generating deblurred image data, wherein generating the deblurred image data includes deblurring, using a deblur circuit of the image sensor, the CIS data (i) based on the EVS data and (ii) internally within the image sensor.

### Claim 22 (depends on 21)

The method of claim 21, further comprising aligning the exposure period and the event accumulation period in time.

### Claim 23 (depends on 22)

The method of claim 22, wherein aligning the exposure period includes resetting the CIS pixels and the one or more EVS pixels such that the exposure period and the event accumulation period have a same start time.

### Claim 24 (depends on 23)

The method of claim 23, further comprising resetting one or more buffers of the deblur circuit such that values stored in the one or more buffers are in a reset state at the same start time of the exposure period and the event accumulation period.

### Claim 25 (depends on 22)

The method of claim 22, wherein aligning the exposure period and the event accumulation period includes aligning the exposure period and the event accumulation period such that the exposure period and the event accumulation period have a same end time.

### Claim 26 (depends on 21)

The method of claim 21, further comprising reading the EVS data out from the one or more EVS pixels as part of a readout operation that cyclically scans the event driven sensing array EVS-pixel-row-by-EVS-pixel-row.

### Claim 27 (depends on 26)

The method of claim 26, wherein the readout operation is configured to spend a preset amount of time at each EVS pixel row of the event driven sensing array to read out corresponding EVS data.

### Claim 28 (depends on 26)

The method of claim 26, wherein the readout operation is configured to (a) spend a preset amount of time at each EVS pixel row of the event driven sensing array having at least one EVS pixel storing unread EVS data, and (b) skip over EVS pixels rows in which no EVS pixel is storing unread EVS data.

### Claim 29 (depends on 21)

The method of claim 21, wherein generating the deblurred image data includes maintaining, over the event accumulation period, a running sum of events included in the EVS data.

### Claim 30 (depends on 29)

The method of claim 29, wherein generating the deblurred image data further includes computing a product of the running sum with a contrast threshold parameter.

### Claim 31 (depends on 30)

The method of claim 30, wherein generating the deblurred image data further includes computing an exponential of the product.

### Claim 32 (depends on 31)

The method of claim 31, wherein generating the deblurred image data further includes computing an integral of the exponential over the event accumulation period.

### Claim 33 (depends on 32)

The method of claim 32, wherein computing the integral includes continuously computing the integral over time during the event accumulation period.

### Claim 34 (depends on 21)

The method of claim 21, wherein generating the deblurred image data includes accumulating the events included in the EVS data, and wherein the method further comprises discarding the EVS data after accumulating the events.

### Claim 35 (depends on 21)

The method of claim 21, wherein generating the deblurred image data includes, at or after an end of the exposure period, reading the CIS data from the CIS pixels to the deblur circuit.

### Claim 36 (depends on 35)

The method of claim 35, wherein the deblur circuit includes a latent frame computation block, and wherein reading the CIS data from the CIS pixels to the deblur circuit includes reading the CIS data from the CIS pixels to the latent frame computation block without first storing the CIS data in a CIS frame buffer.

### Claim 37 (depends on 35)

The method of claim 35, wherein the one or more CIS pixel rows includes a plurality of CIS pixel rows, and wherein reading the CIS data from the CIS pixels of the plurality of CIS pixel rows to the deblur circuit includes reading the CIS data from the CIS pixels of the plurality of CIS pixel rows at a same time.

### Claim 38 (depends on 21)

The method of claim 21, wherein deblurring the CIS data based on the EVS data includes computing a latent frame included in the CIS data.

### Claim 39 (depends on 38)

The method of claim 38, wherein the latent frame corresponds to the intensity of light incident on the CIS pixels at a start time of the exposure period.

### Claim 40 (depends on 38)

The method of claim 38, wherein the latent frame corresponds to the intensity of light incident on the CIS pixels at a time within the exposure period occurring after a start time of the exposure period.

### Claim 41 (depends on 21)

The method of claim 21, further comprising outputting the deblurred image data from the image sensor without outputting the EVS data from the image sensor.

### Claim 42 (depends on 21)

The method of claim 21, wherein:
the CIS data (i) captured using the CIS pixels and (ii) prior to the deblurring, is blurred CIS data; and
the method further comprises outputting the deblurred image data from the image sensor without outputting the blurred CIS data.

### Claim 43 (depends on 21)

The method of claim 21, wherein:
the CIS data (i) captured using the CIS pixels and (ii) prior to the deblurring, is blurred CIS data; and
the method further comprises outputting the EVS data and/or the blurred CIS data.

### Claim 44 (depends on 21)

The method of claim 21, further comprising, at an end of the event accumulation period, disabling the one or more EVS pixels from detecting further events.
