# US 2020/0400795 A1 · Noise aware edge enhancement in a pulsed laser mapping imaging system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0400795 A1
- Document ID: US20200400795A1
- Status: Published application
- Kind: A1
- Application number: 16/743,283
- Filed: 2020-01-15
- Published: 2020-12-24
- Earliest claimed: 2019-06-20
- Assignees: Ethicon, Inc.
- Applicants: Ethicon LLC
- Inventors: Joshua D. Talbert; Donald M. Wichern
- Web page: https://patentgrep.com/patent/US20200400795A1

## CPC

- G01S 7/4865: Time delay measurement, e.g. time-of-flight measurement, time of arrival measurement or determining the exact position of a peak (peak detection in noise, signal conditioning G01S7/487)
- A61B 1/00172: {with means for scanning}
- A61B 1/05: characterised by the image sensor, e.g. camera, being in the distal end portion
- G01B 11/2513: {with several lines being projected in more than one direction, e.g. grids, patterns}
- G01S 7/51: Display arrangements
- G01S 17/89: for mapping or imaging
- G06T 5/73: Deblurring; Sharpening
- G06T 7/13: Edge detection
- H04N 23/555: {for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes}
- H04N 23/56: provided with illuminating means
- A61B 1/043: {for fluorescence imaging}
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215)
- G06T 15/04: Texture mapping
- G06T 2207/00: Indexing scheme for image analysis or image enhancement
- G06T 2207/10024: Color image
- G06T 2207/20182: Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
- G06T 2207/20192: Edge enhancement; Edge preservation
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Laser mapping imaging with reduced fixed pattern noise is disclosed. A method includes actuating an emitter to emit a plurality of pulses of electromagnetic radiation and sensing reflected electromagnetic radiation resulting from the plurality of pulses of electromagnetic radiation with a pixel array of an image sensor to generate a plurality of exposure frames. The method includes applying edge enhancement to edges within an exposure frame of the plurality of exposure frames. The method is such that at least a portion of the plurality of pulses of electromagnetic radiation emitted by the emitter comprises a laser mapping pattern.

## Claims

### Claim 1

A method comprising:
actuating an emitter to emit a plurality of pulses of electromagnetic radiation;
sensing reflected electromagnetic radiation resulting from the plurality of pulses of electromagnetic radiation with a pixel array of an image sensor to generate a plurality of exposure frames; and
applying edge enhancement to edges within an exposure frame of the plurality of exposure frames;
wherein at least a portion of the plurality of pulses of electromagnetic radiation emitted by the emitter comprises a laser mapping pattern.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
detecting the edges within the exposure frame;
retrieving from memory a known conversion gain and an applied sensor gain for the pixel array;
calculating a threshold magnitude of noise acceptable in the exposure frame based on the known conversion gain and the applied sensor gain for the pixel array; and
adjusting a magnitude of the applied edge enhancement based on the threshold magnitude of noise acceptable in the exposure frame.

### Claim 3 (depends on 1)

The method of claim 1, wherein applying the edge enhancement to the edges within the exposure frame comprises:
extracting luminance data from the exposure frame;
detecting the edges within the exposure frame;
applying a gain factor to the detected edges within the image frame to generate modified edge data; and
merging the luminance data and the modified edge data.

### Claim 4 (depends on 1)

The method of claim 1, further comprising calculating expected noise for the plurality of exposure frames generated by the pixel array based on one or more of: a known conversion gain for each pixel within the pixel array, a known applied sensor gain for the pixel array, or a voltage range for a digitizer of an image processing system in electronic communication with the image sensor.

### Claim 5 (depends on 1)

The method of claim 1, wherein applying the edge enhancement comprises applying the edge enhancement in response to the exposure frame comprising more than a threshold magnitude of noise.

### Claim 6 (depends on 1)

The method of claim 1, wherein applying the edge enhancement comprises applying the edge enhancement on a per-pixel basis in response to a pixel comprising more than a threshold magnitude of noise, and wherein the method further comprises determining a per-pixel threshold magnitude of noise for a plurality of pixels in the pixel array based on expected local noise.

### Claim 7 (depends on 1)

The method of claim 1, further comprising detecting the edges within the exposure frame by:
applying a spatial filter to the exposure frame, wherein the spatial filter is a Gaussian filter;
extracting a luminance plane from the exposure frame;
generating a difference plane by subtracting the spatially filtered version of the exposure frame from the luminance plane; and
detecting the edges by identifying local bipolar signals in the difference plane having amplitudes that scale with spatial frequency.

### Claim 8 (depends on 1)

The method of claim 1, further comprising detecting the edges within the exposure frame by:
applying a spatial filter to the exposure frame, wherein the spatial filter is a Gaussian filter; and
applying an edge detect operator kernel to the spatially filtered version of the exposure frame.

### Claim 9 (depends on 1)

The method of claim 1, further comprising calculating a gain factor for governing a magnitude of the edge enhancement applied to the edges within the exposure frame, wherein calculating the gain factor comprises calculating based on:
voltage swing of an analog-to-digital converter (ADC) in electronic communication with the image sensor;
a known conversion gain for the pixel array;
an absolute overall linear gain applied to the image sensor; and
a strength adjuster setting received from a user.

### Claim 10 (depends on 1)

The method of claim 1, further comprising calculating a gain factor for governing a magnitude of the edge enhancement applied to the edges within the exposure frame, wherein calculating the gain factor comprises:
plotting a gain graph by plotting photon transfer curves of noise versus signal for a range of potential gains;
identifying a calibration factor equal to the reciprocal of a gradient within a linear region of the gain graph;
predicting a magnitude of noise expectation based on the calibration factor; and
calculating the gain factor based on the predicted magnitude of noise expectation and a strength adjust setting received from a user.

### Claim 11 (depends on 1)

The method of claim 1, wherein sensing the reflected electromagnetic radiation comprises sensing during a readout period of the pixel array, wherein the readout period is a duration of time when active pixels in the pixel array are read.

### Claim 12 (depends on 1)

The method of claim 1, wherein actuating the emitter comprises actuating the emitter to emit, during a pulse duration, a plurality of sub-pulses of electromagnetic radiation having a sub-duration shorter than the pulse duration.

### Claim 13 (depends on 1)

The method of claim 1, wherein actuating the emitter comprises actuating the emitter to emit two or more wavelengths simultaneously as a single pulse or a single sub-pulse.

### Claim 14 (depends on 1)

The method of claim 1, wherein sensing the reflected electromagnetic radiation comprises generating a fluorescence exposure frame in response to emission by the emitter of a fluorescence excitation pulse, and wherein the method further comprises providing the fluorescence exposure frame to a corresponding fluorescence system that determines a location of a critical tissue structure within a scene based on the fluorescence exposure frame.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
receiving the location of the critical tissue structure from the corresponding fluorescence system;
generating an overlay frame comprising the location of the critical tissue structure; and
combining the overlay frame with a color image frame depicting the scene to indicate the location of the critical tissue structure within the scene.

### Claim 16 (depends on 15)

The method of claim 15, wherein sensing reflected electromagnetic by the pixel array comprises generating a laser mapping exposure frame by sensing reflected electromagnetic radiation resulting from the emitter pulsing the laser mapping pattern, and wherein the method further comprises:
providing the laser mapping exposure frame to a corresponding laser mapping system that determines a topology of the scene and/or dimensions of one or more objects within the scene;
providing the location of the critical tissue structure to the corresponding laser mapping system; and
receiving a topology and/or dimension of the critical tissue structure from the corresponding laser mapping system.

### Claim 17 (depends on 16)

The method of claim 16, wherein the critical tissue structure comprises one or more of a nerve, a ureter, a blood vessel, an artery, a blood flow, or a tumor.

### Claim 18 (depends on 1)

The method of claim 1, further comprising synchronizing timing of the plurality of pulses of electromagnetic radiation to be emitted during a blanking period of the image sensor, wherein the blanking period corresponds to a time between a readout of a last row of active pixels in the pixel array and a beginning of a next subsequent readout of active pixels in the pixel array.

### Claim 19 (depends on 1)

The method of claim 1, wherein sensing the reflected electromagnetic radiation comprises sensing with a first pixel array and a second pixel array such that a three-dimensional image can be generated based on the sensed reflected electromagnetic radiation.

### Claim 20 (depends on 1)

The method of claim 1, wherein actuating the emitter comprises actuating the emitter to emit a sequence of pulses of electromagnetic radiation repeatedly sufficient for generating a video stream comprising a plurality of image frames, wherein each image frame in the video stream comprises data from a plurality of exposure frames, and wherein each of the exposure frames corresponds to a pulse of electromagnetic radiation.

### Claim 21 (depends on 1)

The method of claim 1, wherein at least a portion of the plurality of pulses of electromagnetic radiation emitted by the emitter is an excitation wavelength for fluorescing a reagent, and wherein at least a portion of the reflected electromagnetic radiation sensed by the pixel array is a relaxation wavelength of the reagent.

### Claim 22 (depends on 1)

The method of claim 1, wherein sensing reflected electromagnetic radiation by the pixel array comprises generating a laser mapping exposure frame by sensing reflected electromagnetic radiation resulting from the emitter pulsing the laser mapping pattern, wherein the laser mapping exposure frame comprises information for determining real time measurements comprising one or more of:
a distance from an endoscope to an object;
an angle between an endoscope and the object; or
surface topology information about the object.

### Claim 23 (depends on 22)

The method of claim 22, wherein the laser mapping exposure frame comprises information for determining the real time measurements to an accuracy of less than 10 centimeters.

### Claim 24 (depends on 22)

The method of claim 22, wherein the laser mapping exposure frame comprises information for determining the real time measurements to an accuracy of less than one millimeter.

### Claim 25 (depends on 1)

The method of claim 1, wherein actuating the emitter to emit the plurality of pulses of electromagnetic radiation comprises actuating the emitter to emit a plurality of tool-specific laser mapping patterns for each of a plurality of tools within a scene.

### Claim 26 (depends on 1)

The method of claim 1, wherein the laser mapping pattern emitted by the emitter comprises a first output and a second output that are independent from one another, wherein the first output is for light illumination and the second output is for tool tracking.

### Claim 27

A system comprising:
an emitter for emitting a plurality of pulses of electromagnetic radiation;
an image sensor comprising a pixel array for sensing reflected electromagnetic radiation to generate a plurality of exposure frames; and
one or more processors configurable to execute instructions stored in non-transitory computer readable storage media, the instructions comprising applying edge enhancement to edges within an exposure frame of the plurality of exposure frames;
wherein at least a portion of the pulses of electromagnetic radiation emitted by the emitter comprises a laser mapping pattern.

### Claim 28 (depends on 27)

The system of claim 27, wherein the instructions further comprise:
detecting the edges within the exposure frame;
retrieving from memory a known conversion gain and an applied sensor gain for the pixel array;
calculating a threshold magnitude of noise acceptable in the exposure frame based on the known conversion gain and the applied sensor gain for the pixel array; and
adjusting a magnitude of the applied edge enhancement based on the threshold magnitude of noise acceptable in the exposure frame.

### Claim 29 (depends on 27)

The system of claim 27, wherein the laser mapping pattern emitted by the emitter comprises a plurality of tool-specific laser mapping patterns for each of a plurality of tools within a scene.

### Claim 30 (depends on 27)

The system of claim 27, wherein the laser mapping pattern emitted by the emitter comprises a first output and a second output that are independent from one another, wherein the first output is for light illumination and the second output is for tool tracking.
