# US 11,451,735 B2 · High dynamic range micromirror imaging array systems and methods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,451,735 B2
- Document ID: US11451735B2
- Status: Granted
- Kind: B2
- Application number: 16/677,395
- Filed: 2019-11-07
- Published: 2020-05-21
- Granted: 2022-09-20
- Earliest claimed: 2018-11-20
- Assignees: Teledyne FLIR, LLC
- Applicants: Teledyne FLIR, LLC; Noiseless Imaging Oy Ltd.
- Inventors: Aram Danielyan; Cristóvão Cruz; Alessandro Foi
- Web page: https://patentgrep.com/patent/US11451735B2

## CPC

- H04N 5/7458: {the modulator being an array of deformable mirrors, e.g. digital micromirror device [DMD]}
- G02B 26/0833: {the reflecting element being a micromechanical device, e.g. a MEMS mirror, DMD (G02B26/0825 takes precedence; micromechanical devices in general B81B)}
- H04N 23/20: for generating image signals from infrared radiation only
- H04N 25/57: Control of the dynamic range
- H04N 25/583: with different integration times
- H04N 25/78: Readout circuits for addressed sensors, e.g. output amplifiers or A/D converters
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS

## Abstract

A system comprises a digital micromirror device (DMD), an image sensor comprising an array of sensors operable to capture an image of a scene, a readout integrated circuit (ROIC) operable to generate signals from the sensors corresponding to the captured image of the scene, and an image reconstruction module. The image sensor is operable to capture an image of a scene and comprises an array of photodetector sensors operable to capture an image of a scene at a first frame rate, and a read a readout integrated circuit (ROIC) operable to generate signals from the photodetector sensors corresponding to the captured image of the scene at a second frame rate. A digital micromirror device (DMD) comprising a plurality of micromirrors, each micromirror having at least two physical states, and control circuitry operable to separately control the state of each micromirror, the digital micromirror device operable to receive the image of a scene and reflect the image to the image sensor, whereby the image sensor captures the reflected image of the scene. A processing component is operable to control the operation of the DMD and reconstruct the image from the ROIC.

## Claims

### Claim 1

A system comprising:
an image sensor operable to capture an image of a scene, the image sensor comprising an array of photodetector sensors operable to capture an image of a scene at a first frame rate, and a readout integrated circuit (ROIC) operable to generate signals from the photodetector sensors corresponding to the captured image of the scene;
a digital micromirror device (DMD) comprising a plurality of micromirrors, each micromirror having at least two physical states, and control circuitry operable to separately control the state of each micromirror, the digital micromirror device operable to receive the image of a scene and reflect the image to the image sensor, whereby the image sensor captures the reflected image of the scene; and
a processing component operable to control the operation of the DMD and reconstruct the image from the ROIC, by computing an exposure control map for controlling the micromirrors of the DMD during image capture, capturing via the ROIC a frame of the image received from the DMD, and updating the exposure control map for the next image based at least in part on an analysis of the current frame.

### Claim 2 (depends on 1)

The system of claim 1, wherein the processing component is further operable to: correct pixel intensity in the captured frame;
detect saturated pixels; and
perform interpolation to adjust pixel values.

### Claim 3 (depends on 2)

The system of claim 2, wherein the processing component is further operable to:
perform a denoising procedure on the current frame; and
generate a saturation prediction map for a next image frame, the saturation prediction map based on the current frame;
wherein the saturation prediction map is generated by receiving block-matching information from the denoising processing of the current frame and constructing motion vectors for blocks in the current frame.

### Claim 4 (depends on 3)

The system of claim 3, wherein the processing component is further operable to:
generate the saturation prediction map by predicting positions of reference blocks in the next frame;
aggregate reference blocks in minimum and maximum buffers;
construct the saturation prediction map by comparing minimum/maximum buffers against saturation levels; and
update the exposure control map based on the saturation prediction map.

### Claim 5 (depends on 1)

The system of claim 1, wherein the processing component is further operable to upsample the frame rate using the DMD, including acquiring a frame of an image signal and encoding the frame using the DMD with a binary pseudo-random sequence associated with the increased frame rate.

### Claim 6 (depends on 5)

The system of claim 5, wherein the processing component is further operable to upsample the frame rate using the DMD, performing a reconstruction step including generating an initial estimate using an iterative debinning procedure, including computing a residual and normalizing each element by the bins that contributed to the estimate, and interpolating the result to the objective sampling rate.

### Claim 7 (depends on 6)

The system of claim 6, wherein the processing component is further operable to:
upsample the frame rate using the DMD, performing a reconstruction step including performing residual injection to recover missing high frequencies that were not recovered due to the low-pass nature of the interpolator; and
upsample the frame rate using the DMD, performing a reconstruction step of filtering including performing iterative filtering to remove motion blue and artifacts using a block matching filter.

### Claim 8

A method comprising:
capturing an image of a scene at a first frame rate using an image sensor comprising an array of photodetector sensors to generate signals corresponding to the captured image of the scene;
operating a digital micromirror device (DMD) comprising a plurality of micromirrors, each micromirror having at least two physical states, and wherein operating the DMD comprises separately controlling the state of each micromirror, the DMD operable to receive the image of a scene and reflect the image to the image sensor, whereby the image sensor captures the reflected image of the scene; and
controlling the micromirrors of the DMD in accordance with an exposure control map during image capture;
capturing via the ROIC a frame of the image received from the DMD;
updating the exposure control map for the next image based at least in part on an analysis of the current frame; and
reconstructing the image received from the ROIC.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
capturing a frame of the image received from the DMD;
correcting pixel intensity in the captured frame; and
detecting saturated pixels in the captured frame.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
perform interpolation to adjust pixel values; and
performing a denoising procedure on the current frame.

### Claim 11 (depends on 10)

The method of claim 10, further comprising generating a saturation prediction map for a next image frame, the saturation prediction map based on the current frame.

### Claim 12 (depends on 11)

The method of claim 11, further comprising generating the saturation prediction map by receiving block-matching information from the denoising processing of the current frame; and constructing motion vectors for blocks in the current frame.

### Claim 13 (depends on 12)

The method of claim 12, further comprising generating the saturation prediction map by predicting positions of reference blocks in the next frame and aggregating reference blocks in minimum and maximum buffers.

### Claim 14 (depends on 13)

The method of claim 13, further comprising constructing the saturation prediction map by comparing minimum/maximum buffers against saturation levels; and
updating the exposure control map based at least in part on the saturation prediction map.

### Claim 15 (depends on 14)

The method of claim 14, further comprising upsampling the frame rate using the DMD, including acquiring a frame of an image signal and encoding the frame using the DMD with a binary pseudo-random sequence associated with the increased frame rate.

### Claim 16 (depends on 15)

The method of claim 15, further comprising upsampling the frame rate using the DMD, performing a reconstruction step including generating an initial estimate using an iterative debinning procedure, including computing a residual and normalizing each element by the bins that contributed to the estimate, and interpolating the result to the objective sampling rate.

### Claim 17 (depends on 16)

The method of claim 16, further comprising upsampling the frame rate using the DMD, performing a reconstruction step including performing residual injection to recover missing high frequencies that were not recovered due to the low-pass nature of the interpolator.

### Claim 18 (depends on 17)

The method of claim 17, further comprising upsampling the frame rate using the DMD, performing a reconstruction step of filtering including performing iterative filtering to remove motion blue and artifacts using a block matching filter.