# US 2020/0400571 A1 · Hyperspectral imaging with fixed pattern noise cancellation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0400571 A1
- Document ID: US20200400571A1
- Status: Published application
- Kind: A1
- Application number: 16/735,925
- Filed: 2020-01-07
- Published: 2020-12-24
- Earliest claimed: 2019-06-20
- Assignees: Ethicon, Inc.
- Applicants: Ethicon LLC
- Inventors: Joshua D. Talbert; Donald M. Wichern
- Web page: https://patentgrep.com/patent/US20200400571A1

## CPC

- G01N 21/6456: {Spatial resolved fluorescence measurements; Imaging}
- A47B 88/975: in the form of repositionable partition walls
- A61B 1/000095: {for image enhancement}
- A61B 1/043: {for fluorescence imaging}
- A61B 1/045: Control thereof
- A61B 1/051: {Details of CCD assembly}
- A61B 1/063: {for monochromatic or narrow-band illumination}
- A61B 1/0638: {providing two or more wavelengths}
- A61B 1/0655: {Control therefor}
- A61B 1/07: using light-conductive means, e.g. optical fibres
- G01S 17/89: for mapping or imaging
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/70: Denoising; Smoothing
- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- G09F 15/0006: {planar structures comprising one or more panels}
- H04N 23/125: {Colour sequential image capture, e.g. using a colour wheel}
- H04N 23/555: {for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes}
- H04N 23/56: provided with illuminating means
- H04N 25/633: by using optical black pixels
- H04N 25/673: by using reference sources
- G01N 2021/6439: {with indicators, stains, dyes, tags, labels, marks}
- G01S 7/4802: {using analysis of echo signal for target characterisation; Target signature; Target cross-section}
- G01S 7/483: Details of pulse systems
- G06T 2207/10048: Infrared image
- G06T 2207/10064: Fluorescence image
- G06T 2207/10068: Endoscopic image
- G06T 2207/20224: Image subtraction
- G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q)
- A47B: TABLES; DESKS; OFFICE FURNITURE; CABINETS; DRAWERS; GENERAL DETAILS OF FURNITURE (furniture jointing F16B)
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G09F: DISPLAYING; ADVERTISING; SIGNS; LABELS OR NAME-PLATES; SEALS
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Hyperspectral imaging with reduced fixed pattern noise is disclosed. A method includes actuating an emitter to emit a plurality of pulses of electromagnetic radiation and sensing reflected electromagnetic radiation resulting from the plurality of pulses of electromagnetic radiation with a pixel array of an image sensor. The method includes reducing fixed pattern noise in an exposure frame by subtracting a reference frame from the exposure frame. The method is such that at least a portion of the plurality of pulses of electromagnetic radiation emitted by the emitter comprises one or more of electromagnetic radiation having a wavelength from about 513 nm to about 545 nm, from about 565 nm to about 585 nm, or from about 900 nm to about 1000 nm.

## Claims

### Claim 1

A method comprising:
actuating an emitter to emit a plurality of pulses of electromagnetic radiation;
sensing reflected electromagnetic radiation resulting from the plurality of pulses of electromagnetic radiation with a pixel array of an image sensor to generate a plurality of exposure frames;
generating a reference frame for use in removing fixed pattern noise from the plurality of exposure frames, wherein the reference frame is based on dark frame data captured when the emitter is not emitting electromagnetic radiation; and
reducing fixed pattern noise in an exposure frame of the plurality of exposure frames by subtracting the reference frame from the exposure frame;
wherein at least a portion of the plurality of pulses of electromagnetic radiation emitted by the emitter comprises one or more of:
electromagnetic radiation having a wavelength from about 513 nm to about 545 nm;
electromagnetic radiation having a wavelength from about 565 nm to about 585 nm; or
electromagnetic radiation having a wavelength from about 900 nm to about 1000 nm.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
stopping the emitter from pulsing electromagnetic radiation; and
sensing the dark frame data with the pixel array when the emitter is not emitting electromagnetic radiation.

### Claim 3 (depends on 1)

The method of claim 1, wherein the plurality of pulses of electromagnetic radiation are emitted in a pattern of varying wavelengths of electromagnetic radiation, and wherein the emitter repeats the pattern of varying wavelengths of electromagnetic radiation.

### Claim 4 (depends on 3)

The method of claim 3, wherein the pattern of varying wavelengths of electromagnetic radiation comprises a zero-emission period wherein the emitter does not pulse electromagnetic radiation, and wherein the method further comprises sensing the dark frame data with the pixel array during the zero-emission period.

### Claim 5 (depends on 1)

The method of claim 1, further comprising sensing a plurality of dark exposure frames with the pixel array when the emitter is not emitting electromagnetic radiation, wherein each of the plurality of dark exposure frames comprises dark frame data.

### Claim 6 (depends on 5)

The method of claim 5, wherein the plurality of dark exposure frames are interspersed within the plurality of exposure frames.

### Claim 7 (depends on 5)

The method of claim 5, wherein generating the reference frame comprises generating a plurality of reference frames based on the plurality of dark exposure frames, and wherein the method further comprises enhancing precision of at least one reference frame of the plurality of reference frames by sampling subsequent dark frame data from subsequent dark exposure frames.

### Claim 8 (depends on 7)

The method of claim 7, wherein enhancing precision of the at least one reference frame comprises accounting for the subsequent dark frame data from the subsequent dark exposure frames using exponential smoothing.

### Claim 9 (depends on 1)

The method of claim 1, wherein generating the reference frame comprises interpolating two or more instances of dark frame data from two or more dark exposure frames captured at different integration times.

### Claim 10 (depends on 1)

The method of claim 1, wherein sensing the reflected electromagnetic radiation comprises sensing during a readout period of the pixel array, wherein the readout period is a duration of time when active pixels in the pixel array are read.

### Claim 11 (depends on 1)

The method of claim 1, wherein actuating the emitter comprises actuating the emitter to emit, during a pulse duration, a plurality of sub-pulses of electromagnetic radiation having a sub-duration shorter than the pulse duration.

### Claim 12 (depends on 1)

The method of claim 1, wherein actuating the emitter comprises actuating the emitter to emit two or more wavelengths simultaneously as a single pulse or a single sub-pulse.

### Claim 13 (depends on 1)

The method of claim 1, wherein sensing the reflected electromagnetic radiation comprises generating a hyperspectral exposure frame based on a hyperspectral pulse emitted by the emitter, and wherein the method further comprises providing the hyperspectral exposure frame to a corresponding system that determines a location of a critical tissue structure within a scene based on the hyperspectral exposure frame.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
receiving the location of the critical tissue structure from the corresponding system;
generating an overlay frame comprising the location of the critical tissue structure; and
combining the overlay frame with a color image frame depicting the scene to indicate the location of the critical tissue structure within the scene.

### Claim 15 (depends on 14)

The method of claim 14, wherein the critical tissue structure comprises one or more of a nerve, a ureter, a blood vessel, an artery, a blood flow, or a tumor.

### Claim 16 (depends on 1)

The method of claim 1, further comprising synchronizing timing of the plurality of pulses of electromagnetic radiation to be emitted during a blanking period of the image sensor, wherein the blanking period corresponds to a time between a readout of a last row of active pixels in the pixel array and a beginning of a next subsequent readout of active pixels in the pixel array.

### Claim 17 (depends on 1)

The method of claim 1, wherein sensing the reflected electromagnetic radiation comprises sensing with a first pixel array and a second pixel array such that a three-dimensional image can be generated based on the sensed reflected electromagnetic radiation.

### Claim 18 (depends on 1)

The method of claim 1, wherein actuating the emitter comprises actuating the emitter to emit a sequence of pulses of electromagnetic radiation repeatedly sufficient for generating a video stream comprising a plurality of image frames, wherein each image frame in the video stream comprises data from a plurality of exposure frames, and wherein each of the exposure frames corresponds to a pulse of electromagnetic radiation.

### Claim 19 (depends on 1)

The method of claim 1, wherein actuating the emitter comprises actuating the emitter to emit a red wavelength, a green wavelength, a blue wavelength, and a hyperspectral wavelength such that reflected electromagnetic radiation sensed by the pixel array corresponding to each of the red wavelength, the green wavelength, the blue wavelength, and the hyperspectral wavelength can be processed to generate a Red-Green-Blue (RGB) image comprising an overlay of hyperspectral imaging data, wherein the hyperspectral wavelength of electromagnetic radiation comprises:
the electromagnetic radiation having the wavelength from about 513 nm to about 545 nm and the electromagnetic radiation having the wavelength from about 900 nm to about 1000 nm; or
the electromagnetic radiation having the wavelength from about 565 nm to about 585 nm and the electromagnetic radiation having the wavelength from about 900 nm to about 1000 nm.

### Claim 20

A system comprising:
an emitter for emitting a plurality of pulses of electromagnetic radiation;
an image sensor comprising a pixel array for sensing reflected electromagnetic radiation to generate a plurality of exposure frames; and
one or more processors configurable to execute instructions stored in non-transitory computer readable storage media, the instructions comprising:
generating a reference frame for use in removing fixed pattern noise from the plurality of exposure frames, wherein the reference frame is based on dark frame data captured when the emitter is not emitting electromagnetic radiation; and
reducing fixed pattern noise in an exposure frame of the plurality of exposure frames by subtracting the reference frame from the exposure frame;
wherein at least a portion of the pulses of electromagnetic radiation emitted by the emitter comprises one or more of:
electromagnetic radiation having a wavelength from about 513 nm to about 545 nm;
electromagnetic radiation having a wavelength from about 565 nm to about 585 nm; or
electromagnetic radiation having a wavelength from about 900 nm to about 1000 nm.

### Claim 21 (depends on 20)

The system of claim 20, wherein the instructions further comprise:
stopping the emitter from pulsing electromagnetic radiation; and
sensing the dark frame data with the pixel array when the emitter is not emitting electromagnetic radiation.

### Claim 22 (depends on 20)

The system of claim 20, wherein the plurality of pulses of electromagnetic radiation are emitted in a pattern of varying wavelengths of electromagnetic radiation, and wherein the emitter repeats the pattern of varying wavelengths of electromagnetic radiation.

### Claim 23 (depends on 22)

The system of claim 22, wherein the pattern of varying wavelengths of electromagnetic radiation comprises a zero-emission period wherein the emitter does not pulse electromagnetic radiation, and wherein the method further comprises sensing the dark frame data with the pixel array during the zero-emission period.

### Claim 24 (depends on 20)

The system of claim 20, wherein the pixel array senses a plurality of dark exposure frames over a plurality of instances when the emitter is not emitting electromagnetic radiation, wherein each of the plurality of dark exposure frames comprises dark frame data.

### Claim 25 (depends on 24)

The system of claim 24, wherein the instructions are such that generating the reference frame comprises generating a plurality of reference frames based on the plurality of dark exposure frames, and wherein the method further comprises enhancing precision of at least one reference frame of the plurality of reference frames by sampling subsequent dark frame data from subsequent dark exposure frames.

### Claim 26 (depends on 20)

The system of claim 20, wherein at least a portion of the pulses of electromagnetic radiation emitted by the emitter results in a hyperspectral exposure frame created by the image sensor, and wherein the instructions further comprise providing the hyperspectral exposure frame to a corresponding system that determines a location of a critical tissue structure within a scene based on the hyperspectral exposure frame.

### Claim 27 (depends on 26)

The system of claim 26, wherein the instructions further comprise:
receiving the location of the critical tissue structure from the corresponding system;
generating an overlay frame comprising the location of the critical tissue structure; and
combining the overlay frame with a color image frame depicting the scene to indicate the location of the critical tissue structure within the scene.

### Claim 28 (depends on 27)

The system of claim 27, wherein the critical tissue structure comprises one or more of a nerve, a ureter, a blood vessel, an artery, a blood flow, or a tumor.

### Claim 29 (depends on 20)

The system of claim 20, wherein the image sensor comprises a first image sensor and a second image sensor such that the image sensor can generate a three-dimensional image.

### Claim 30 (depends on 20)

The system of claim 20, wherein at least a portion of the plurality of pulses of electromagnetic radiation comprise a red wavelength, a green wavelength, a blue wavelength, and a hyperspectral wavelength such that reflected electromagnetic radiation sensed by the pixel array corresponding to each of the red wavelength, the green wavelength, the blue wavelength, and the hyperspectral wavelength can be processed to generate a Red-Green-Blue (RGB) image comprising an overlay of hyperspectral imaging data, wherein the hyperspectral wavelength of electromagnetic radiation comprises:
the electromagnetic radiation having the wavelength from about 513 nm to about 545 nm and the electromagnetic radiation having the wavelength from about 900 nm to about 1000 nm; or
the electromagnetic radiation having the wavelength from about 565 nm to about 585 nm and the electromagnetic radiation having the wavelength from about 900 nm to about 1000 nm.
