# US 2020/0400570 A1 · Super resolution and color motion artifact correction in a pulsed hyperspectral, fluorescence, and laser mapping imaging system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0400570 A1
- Document ID: US20200400570A1
- Status: Published application
- Kind: A1
- Application number: 16/719,242
- Filed: 2019-12-18
- Published: 2020-12-24
- Earliest claimed: 2019-06-20
- Assignees: Ethicon, Inc.
- Applicants: Ethicon LLC
- Inventors: Joshua D. Talbert; Donald M. Wichern
- Web page: https://patentgrep.com/patent/US20200400570A1

## CPC

- G06T 5/73: Deblurring; Sharpening
- A61B 1/000095: {for image enhancement}
- A61B 1/043: {for fluorescence imaging}
- A61B 1/045: Control thereof
- A61B 1/051: {Details of CCD assembly}
- A61B 1/063: {for monochromatic or narrow-band illumination}
- A61B 1/0638: {providing two or more wavelengths}
- A61B 1/0655: {Control therefor}
- A61B 1/07: using light-conductive means, e.g. optical fibres
- G01N 21/6456: {Spatial resolved fluorescence measurements; Imaging}
- G01S 7/4815: {using multiple transmitters}
- G01S 7/4816: {of receivers alone}
- G01S 7/484: Transmitters
- G01S 17/89: for mapping or imaging
- G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403)
- G06T 3/4061: by injecting details from different spectral ranges
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- H04N 23/11: for generating image signals from visible and infrared light wavelengths
- H04N 23/125: {Colour sequential image capture, e.g. using a colour wheel}
- H04N 23/555: {for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes}
- H04N 23/56: provided with illuminating means
- H04N 23/665: {involving internal camera communication with the image sensor, e.g. synchronising or multiplexing SSIS control signals}
- H04N 23/6811: {based on the image signal}
- H04N 23/683: {performed by a processor, e.g. controlling the readout of an image memory}
- H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio
- G01N 2021/6439: {with indicators, stains, dyes, tags, labels, marks}
- G01S 7/483: Details of pulse systems
- G06T 2207/10024: Color image
- G06T 2207/10064: Fluorescence image
- G06T 2207/10068: Endoscopic image
- G06T 2207/10152: Varying illumination
- G06T 2207/20201: Motion blur correction
- G06T 2207/20221: Image fusion; Image merging
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q)
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Super resolution and color motion artifact correction in a pulsed hyperspectral, fluorescence, and laser mapping imaging system. A method includes actuating an emitter to emit pulses of electromagnetic radiation and sensing reflected electromagnetic radiation with a pixel array of an image sensor. The method includes detecting motion across two or more sequential exposure frames, compensating for the detected motion, and combining the two or more sequential exposure frames to generate an image frame. The method is such that at least a portion of the pulses of electromagnetic radiation emitted by the emitter comprises one or more of: electromagnetic radiation having a wavelength from about 513 nm to about 545 nm, from about 565 nm to about 585 nm, from about 900 nm to about 1000 nm, an excitation wavelength of electromagnetic radiation that causes a reagent to fluoresce, or a laser mapping pattern.

## Claims

### Claim 1

A method comprising:
actuating an emitter to emit a plurality of pulses of electromagnetic radiation;
sensing reflected electromagnetic radiation resulting from the plurality of pulses of electromagnetic radiation with a pixel array of an image sensor to generate a plurality of exposure frames;
detecting motion across two or more sequential exposure frames of the plurality of exposure frames;
compensating for the detected motion; and
combining the two or more sequential exposure frames to generate an image frame;
wherein at least a portion of the pulses of electromagnetic radiation emitted by the emitter comprises one or more of:
electromagnetic radiation having a wavelength from about 513 nm to about 545 nm;
electromagnetic radiation having a wavelength from about 565 nm to about 585 nm;
electromagnetic radiation having a wavelength from about 900 nm to about 1000 nm;
an excitation wavelength of electromagnetic radiation that causes a reagent to fluoresce; or
a laser mapping pattern.

### Claim 2 (depends on 1)

The method of claim 1, wherein compensating for the detected motion comprises:
upscaling a first exposure frame of the two or more sequential exposure frames using interpolation to generate a first upscaled frame;
upscaling the first exposure frame without using interpolation to generate a second upscaled frame, wherein the second upscaled frame comprises a first set of empty pixels; and
filling in the first set of empty pixels of the second upscaled frame with pixel data from the first upscaled frame.

### Claim 3 (depends on 2)

The method of claim 2, wherein compensating for the detected motion further comprises:
upscaling a second exposure frame of the two or more sequential exposure frames to generate a third upscaled frame; and
filling in a second set of empty pixels in the second upscaled frame with pixel data from the third upscaled frame.

### Claim 4 (depends on 1)

The method of claim 1, wherein the two or more sequential exposure frames comprises a red exposure frame, a green exposure frame, and a blue exposure frame, and wherein combining the two or more sequential exposure frames to generate the image frame comprises generating a Red Green Blue (“RGB”) image frame.

### Claim 5 (depends on 1)

The method of claim 1, wherein the two or more sequential exposure frames comprises a luminance (Y) exposure frame, a chrominance blue (Cb) exposure frame, and a chrominance red (Cr) exposure frame, and wherein combining the two or more sequential exposure frames to generate the image frame comprises generating a YCbCr image frame.

### Claim 6 (depends on 1)

The method of claim 1, wherein sensing the reflected electromagnetic radiation comprises:
generating a first exposure frame based on a pulse of electromagnetic radiation of a first color partition;
generating a second exposure frame based on a pulse of electromagnetic radiation of a second color partition; and
generating a third exposure frame based on a pulse of electromagnetic radiation of the first color partition;
wherein the second exposure frame is captured between the first exposure frame and the third exposure frame;
wherein detecting motion across the two or more sequential exposure frames comprises calculating a relative motion estimate based on the first exposure frame and the third exposure frame using block matching; and
wherein compensating for the detected motion comprises generating a motion compensated frame for the second exposure frame based on the relative motion estimate.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
determining a first motion vector for the first exposure frame and a second motion vector for the second exposure frame; and
shifting a block of pixels in the first exposure frame by the first motion vector.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
performing bilinear interpolation on luminance data in the two or more sequential exposure frames to generate a first upscaled dataset;
performing bicubic interpolation on the luminance data to generate a second upscaled dataset; and
calculating a baseline with no interpolation of the luminance data to generate a third upscaled dataset.

### Claim 9 (depends on 8)

The method of claim 8, wherein detecting motion across two or more sequential exposure frames comprises one or more of:
segmenting data sensed by the pixel array into segments of pixels and nearest neighboring exposure frames;
shifting each segment of pixels in the x direction and comparing with a neighboring exposure frame at a same resolution to identify motion of an object being imaged in the x direction;
shifting each segment of pixels in the x direction in sub-pixel increments and comparing to the first upscaled dataset to identify motion of the object being imaged in the x direction with increased precision;
shifting each segment of pixels in the y direction and comparing with a neighboring exposure frame to identify motion of an object being imaged in the y direction; or
shifting each segment of pixels in the y direction in sub-pixel increments and comparing to the first upscaled dataset to identify motion of the object being imaged in the y direction with increased precision.

### Claim 10 (depends on 1)

The method of claim 1, wherein sensing the reflected electromagnetic radiation comprises sensing during a readout period of the pixel array, wherein the readout period is a duration of time when active pixels in the pixel array are read.

### Claim 11 (depends on 1)

The method of claim 1, wherein actuating the emitter comprises actuating the emitter to emit, during a pulse duration, a plurality of sub-pulses of electromagnetic radiation having a sub-duration shorter than the pulse duration.

### Claim 12 (depends on 1)

The method of claim 1, wherein actuating the emitter comprises actuating the emitter to emit two or more wavelengths simultaneously as a single pulse or a single sub-pulse.

### Claim 13 (depends on 1)

The method of claim 1, wherein actuating the emitter comprises actuating the emitter to pulse the laser mapping pattern at a duration and frequency such that the laser mapping pattern is not visible to a user of the system.

### Claim 14 (depends on 1)

The method of claim 1, wherein sensing the reflected electromagnetic radiation comprises generating a fluorescence exposure frame, and wherein the method further comprises providing the fluorescence exposure frame to a corresponding fluorescence system that determines a location of a critical tissue structure within a scene based on the fluorescence exposure frame.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
receiving the location of the critical tissue structure from the corresponding fluorescence system;
generating an overlay frame comprising the location of the critical tissue structure; and
combining the overlay frame with a color image frame depicting the scene to indicate the location of the critical tissue structure within the scene.

### Claim 16 (depends on 15)

The method of claim 15, wherein sensing reflected electromagnetic by the pixel array comprises sensing reflected electromagnetic radiation resulting from the laser mapping pattern to generate a laser mapping exposure frame, and wherein the method further comprises:
providing the laser mapping exposure frame to a corresponding laser mapping system that determines a topology of the scene and/or dimensions of one or more objects within the scene;
providing the location of the critical tissue structure to the corresponding laser mapping system; and
receiving a topology and/or dimension of the critical tissue structure from the corresponding laser mapping system.

### Claim 17 (depends on 16)

The method of claim 16, wherein the critical tissue structure comprises one or more of a nerve, a ureter, a blood vessel, an artery, a blood flow, or a tumor.

### Claim 18 (depends on 1)

The method of claim 1, further comprising synchronizing timing of the plurality of pulses of electromagnetic radiation to be emitted during a blanking period of the image sensor, wherein the blanking period corresponds to a time between a readout of a last row of active pixels in the pixel array and a beginning of a next subsequent readout of active pixels in the pixel array.

### Claim 19 (depends on 1)

The method of claim 1, wherein the two or more sequential exposure frames are captured sequentially in time based on two or more pulses of electromagnetic radiation emitted by the emitter sequentially in time.

### Claim 20 (depends on 1)

The method of claim 1, wherein sensing the reflected electromagnetic radiation comprises sensing with a first pixel array and a second pixel array such that a three-dimensional image can be generated based on the sensed reflected electromagnetic radiation.

### Claim 21 (depends on 1)

The method of claim 1, wherein actuating the emitter comprises actuating the emitter to emit a sequence of pulses of electromagnetic radiation repeatedly sufficient for generating a video stream comprising a plurality of image frames, wherein each image frame in the video stream comprises data from a plurality of exposure frames, and wherein each of the exposure frames corresponds to a pulse of electromagnetic radiation.

### Claim 22 (depends on 1)

The method of claim 1, wherein sensing reflected electromagnetic radiation by the pixel array comprises generating a laser mapping exposure frame by sensing reflected electromagnetic radiation resulting from the emitter pulsing the laser mapping pattern, wherein the laser mapping exposure frame comprises information for determining real time measurements comprising one or more of:
a distance from an endoscope to an object;
an angle between an endoscope and the object; or
surface topology information about the object.

### Claim 23 (depends on 22)

The method of claim 22, wherein the laser mapping exposure frame comprises information for determining the real time measurements to an accuracy of less than 10 centimeters.

### Claim 24 (depends on 22)

The method of claim 22, wherein the laser mapping exposure frame comprises information for determining the real time measurements to an accuracy of less than one millimeter.

### Claim 25 (depends on 1)

The method of claim 1, wherein actuating the emitter to emit the plurality of pulses of electromagnetic radiation comprises actuating the emitter to emit a plurality of tool-specific laser mapping patterns for each of a plurality of tools within a scene.

### Claim 26 (depends on 1)

The method of claim 1, wherein the laser mapping pattern emitted by the emitter comprises a first output and a second output that are independent from one another, wherein the first output is for light illumination and the second output is for tool tracking.

### Claim 27 (depends on 1)

The method of claim 1, wherein at least a portion of the plurality of pulses of electromagnetic radiation comprise a hyperspectral emission comprising one of:
electromagnetic radiation having a wavelength from about 513 nm to about 545 nm and electromagnetic radiation having a wavelength from about 900 nm to about 1000 nm; or
electromagnetic radiation having a wavelength from about 565 nm to about 585 nm and electromagnetic radiation having a wavelength from about 900 nm to about 1000 nm;
wherein sensing reflected electromagnetic radiation by the pixel array comprises generating a hyperspectral exposure frame based on the hyperspectral emission.

### Claim 28 (depends on 27)

The method of claim 27, further comprising:
providing the hyperspectral exposure frame to a corresponding hyperspectral system that determines a location of a critical tissue structure based on the hyperspectral exposure frame;
receiving the location of the critical tissue structure from the corresponding hyperspectral system;
generating an overlay frame comprising the location of the critical tissue structure; and
combining the overlay frame with a color image frame depicting the scene to indicate the location of the critical tissue structure within the scene.

### Claim 29 (depends on 28)

The method of claim 28, wherein sensing reflected electromagnetic radiation by the pixel array comprises generating a laser mapping exposure frame based on emission of the laser mapping pattern, and wherein the method further comprises:
providing the laser mapping exposure frame to a corresponding laser mapping system that determines a topology of the scene and/or dimensions of one or more objects within the scene;
providing the location of the critical tissue structure to the corresponding laser mapping system; and
receiving a topology and/or dimension of the critical tissue structure from the corresponding laser mapping system.

### Claim 30 (depends on 29)

The method of claim 29, wherein the critical tissue structure is one or more of a nerve, a ureter, a blood vessel, an artery, a blood flow, cancerous tissue, or a tumor.
