# US 11,998,220 B2 · Systems and devices for endoscopic procedure analysis

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,998,220 B2
- Document ID: US11998220B2
- Status: Granted
- Kind: B2
- Application number: 17/396,681
- Filed: 2021-08-07
- Published: 2022-02-10
- Granted: 2024-06-04
- Earliest claimed: 2020-08-10
- Assignees: Kunnskap Medical, LLC
- Applicants: Kunnskap Medical, LLC
- Inventors: Michael C. Holmstrom
- Web page: https://patentgrep.com/patent/US11998220B2

## CPC

- A61B 17/1604: {Chisels; Rongeurs; Punches; Stamps}
- A61B 1/00006: {of control signals}
- A61B 1/00009: {of image signals during a use of endoscope}
- A61B 1/00013: {using optical means}
- A61B 1/0005: {combining images e.g. side-by-side, superimposed or tiled}
- A61B 1/015: Control of fluid supply or evacuation
- A61B 1/04: combined with photographic or television appliances
- A61B 1/045: Control thereof
- A61B 1/317: for bones or joints, e.g. osteoscopes or arthroscopes
- A61B 17/00234: {for minimally invasive surgery (A61B17/0218, A61B17/0469, A61B17/12013, A61B17/1285, A61B17/29, A61B17/320016 take precedence)}
- A61B 18/148: {having a short, rigid shaft for accessing the inner body transcutaneously, e.g. for neurosurgery or arthroscopy}
- A61B 18/22: the beam being directed along or through a flexible conduit, e.g. an optical fibre; {Couplings or} hand-pieces therefor
- A61B 34/25: {User interfaces for surgical systems}
- A61M 13/003: {Blowing gases other than for carrying powders, e.g. for inflating, dilating or rinsing}
- A61M 13/006: {with gas recirculation}
- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06T 7/0012: {Biomedical image inspection}
- G16H 15/00: ICT specially adapted for medical reports, e.g. generation or transmission thereof
- G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS
- G16H 30/40: for processing medical images, e.g. editing
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- H04N 7/183: {for receiving images from a single remote source}
- H04N 7/185: {from a mobile camera, e.g. for remote control}
- H04N 23/52: Elements optimising image sensor operation, e.g. for electromagnetic interference [EMI] protection or temperature control by heat transfer or cooling elements
- H04N 23/56: provided with illuminating means
- H04N 23/61: based on recognised objects
- H04N 23/74: by influencing the scene brightness using illuminating means
- A61B 18/1482: {having a long rigid shaft for accessing the inner body transcutaneously in minimal invasive surgery, e.g. laparoscopy}
- A61B 18/1492: {having a flexible, catheter-like structure, e.g. for heart ablation (A61B18/1477 takes precedence)}
- A61B 90/30: Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure
- A61B 90/36: {Image-producing devices or illumination devices not otherwise provided for}
- A61B 2017/00367: {Details of actuation of instruments, e.g. relations between pushing buttons, or the like, and activation of the tool, working tip, or the like}
- A61B 2017/1602: {Mills}
- A61B 2018/00577: {Ablation}
- A61B 2018/00589: {Coagulation}
- A61B 2018/00595: {Cauterization}
- A61B 2018/00601: {Cutting}
- A61B 2018/00607: {Coagulation and cutting with the same instrument}
- A61B 2018/00642: {with feedback, i.e. closed loop control}
- A61B 2018/00702: {Power or energy}
- A61B 2018/00744: {Fluid flow}
- A61B 2018/00982: {combined with or comprising means for visual or photographic inspections inside the body, e.g. endoscopes}
- A61B 2018/167: {Passive electrodes capacitively coupled to the skin}
- A61B 2090/309: {using white LEDs}
- A61B 2217/005: with suction drainage system
- A61B 2217/007: with irrigation system
- A61B 2218/002: Irrigation
- A61B 2218/007: Aspiration
- A61B 2218/008: for smoke evacuation
- A61M 2202/0225: Carbon oxides, e.g. Carbon dioxide
- A61M 2205/3334: Measuring or controlling the flow rate
- A61M 2205/3344: Measuring or controlling pressure at the body treatment site
- A61M 2205/502: User interfaces, e.g. screens or keyboards
- A61M 2205/587: Lighting arrangements
- A61M 2205/6072: Bar codes
- G05D 7/0676: {by action on flow sources (G05D7/0688, G05D7/0694 take precedence)}
- G05D 16/2066: {using controlling means acting on the pressure source}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/10068: Endoscopic image
- G06T 2207/10152: Varying illumination
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30004: Biomedical image processing
- G06T 2207/30008: Bone
- G06T 2207/30028: Colon; Small intestine
- G06T 2207/30061: Lung
- G06T 2207/30092: Stomach; Gastric
- G06T 2207/30168: Image quality inspection
- H04N 23/555: {for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes}
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- A61M: DEVICES FOR INTRODUCING MEDIA INTO, OR ONTO, THE BODY (introducing media into or onto the bodies of animals A61D7/00; means for inserting tampons A61F13/26; devices for administering food or medicines orally A61J; containers for collecting, storing or administering blood or medical fluids A61J1/05); DEVICES FOR TRANSDUCING BODY MEDIA OR FOR TAKING MEDIA FROM THE BODY (surgery A61B; chemical aspects of surgical articles A61L); DEVICES FOR PRODUCING OR ENDING SLEEP OR STUPOR {(Electrotherapy, e.g. producing anaesthesia by the use of alternating or intermittent currents A61N1/36021)}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES

## Abstract

Endoscopic image analysis, endoscopic procedure analysis, and/or component control systems, methods and techniques are disclosed that can analyze images of an endoscopic system and/or affect an endoscopic system to enhance operation, user and patient experience, and usability of image data and other case data.

## Claims

### Claim 1

An endoscopic procedure analysis system comprising:
a data capture engine to capture endoscopic case data at intervals during an endoscopic case, the endoscopic case data including state data for one or more components of the endoscopic system, the state data captured by the data capture engine from a control engine at one or more intervals during the endoscopic case and image analysis data generated from analysis of one or more images by an image analysis engine, the one or more images from image data captured by a camera of a view provided by an endoscope of an endoscopic system during the endoscopic case, the analysis including identification of one or more characteristics of the one or more images; and
an endoscopic procedure analysis engine to:
access a set of data from stored endoscopic case data captured by the data capture engine for one or more endoscopic procedures, wherein the set of data includes the state data and the image analysis data;
analyze the set of data, comprising analyzing a combination of the state data and the image analysis data: and
generate additional data based on the analysis of the set of data,
wherein generating the additional data comprises generating additional data regarding timing of the endoscopic case by:
determining a start time by analyzing the image analysis data to determine a first transition based at least in part on a first image having a first characteristic to a second image having a second characteristic;
determining an end time by analyzing the image analysis data to determine a transition from a third image having the second characteristic to a fourth image having a different characteristic, wherein the second characteristic corresponds to when the endoscope was in a body of the patient; and
determining a duration from the start time and the end time,
wherein generating the additional data comprises generating additional data regarding average pump pressure in the endoscopic case by:
identifying in-body images from the image data having the second characteristic corresponding to when the endoscope was in the body of the patient;
determining a pressure of the pump at times corresponding to the in-body images; and
determining an average pressure of the pump during the endoscopic case based on the pressure of the pump at the times corresponding to the in-body images.

### Claim 2 (depends on 1)

The system of claim 1, further comprising:
one or more central datastores to collect endoscopic case data in aggregate with other endoscopic case data from one or more other data capture engines to store an aggregate of endoscopic case data,
wherein the endoscopic procedure analysis engine accesses the set of data from the aggregate of endoscopic case data.

### Claim 3 (depends on 1)

The system of claim 1, further comprising:
the image analysis engine to analyze the one or more images to identify one or more characteristics and generate the image analysis data.

### Claim 4 (depends on 3)

The system of claim 3, further comprising:
an interface to couple to one or more of the camera and an image processing component attached to the camera and to receive the image data captured by the camera.

### Claim 5 (depends on 3)

The system of claim 3, further comprising:
an image data repository interface to couple to a memory storing the image data captured by the camera.

### Claim 6 (depends on 1)

The system of claim 1, wherein the endoscopic case data further includes one or more of:
case images from the image data captured by the camera;
a reference to the case images as stored in a separate location; case videos from the image data captured by the camera; and a reference to the case videos as stored in a separate location.

### Claim 7 (depends on 1)

The system of claim 1, wherein the endoscopic case data further comprises one or more of:
case identifying data;
external data received at the endoscopic system from a source external to the endoscopic system; and
input from a user.

### Claim 8 (depends on 1)

The system of claim 1, wherein the data capture engine captures endoscopic case data at intervals during the endoscopic procedure by one or more of flagging, tagging, organizing, and structuring image analysis data to correlate endoscopic case data for a point in time.

### Claim 9 (depends on 1)

The system of claim 1, wherein the endoscopic procedure analysis engine analyzes the set of data using one or more algorithms.

### Claim 10 (depends on 1)

The system of claim 1, wherein the endoscopic procedure analysis engine generates additional data regarding tissue within the patient as viewed during the endoscopic case.

### Claim 11

An endoscopic procedure analysis component comprising:
an image data interface to couple to an endoscopic system to receive endoscopic image data captured by a camera of the endoscopic system during an endoscopic procedure performed utilizing the endoscopic system, the image data of a view provided by an endoscope of the endoscopic system, the endoscope to access an area to be visualized within a patient;
an image analysis engine to:
analyze one or more images from the endoscopic image data to identify one or more characteristics of each of the one or more images; and
generate image analysis data including the one or more characteristics of each of the one or more images;
a data capture engine to capture endoscopic case data at intervals during the endoscopic procedure, the endoscopic case data including state data for one or more components of the endoscopic system, the state data captured by the data capture engine from a control engine at one or more intervals during the endoscopic case and the image analysis data; and
an endoscopic procedure analysis engine to:
access an aggregate of endoscopic case data for one or more endoscopic procedures, wherein the set of data includes the state data and the image analysis data;
analyze a set of data from the aggregate of endoscopic case data utilizing one or more algorithms that consider characteristics of images in the image analysis data, wherein analyzing the set of data comprises analyzing a combination of the state data and the image analysis data; and
derive additional data based on the analysis of the set of data,
wherein deriving the additional data comprises generating additional data regarding timing of the endoscopic case by:
determining a start time by analyzing the image analysis data to determine a first transition from a first image having a first characteristic to a second image having a second characteristic;
determining an end time by analyzing the image analysis data to determine a transition from a third image having the second characteristic to a fourth image having a different characteristic, wherein the second characteristic corresponds to when the endoscope was in a body of the patient; and
determining a duration from the start time and the end time,
wherein generating the additional data comprises generating additional data regarding average pump pressure in the endoscopic case by:
identifying in-body images from the image data having the second characteristic corresponding to when the endoscope was in the body of the patient;
determining a pressure of the pump at times corresponding to the in-body images; and
determining an average pressure of the pump during the endoscopic case based on the pressure of the pump at the times corresponding to the in-body images.

### Claim 12 (depends on 11)

The component of claim 11, wherein the data capture engine collects the endoscopic case data in aggregate to store the aggregate of endoscopic case data.

### Claim 13 (depends on 11)

The component of claim 11, further comprising:
a central interface to couple with one or more central datastores that collects the endoscopic case data in aggregate with other endoscopic case data from one or more other data capture engines to store the aggregate of endoscopic case data.

### Claim 14 (depends on 11)

The component of claim 11, further comprising:
one or more central datastores to collect endoscopic case data in aggregate with other endoscopic case data from one or more other data capture engines to store the aggregate of endoscopic case data in the one or more central datastores.

### Claim 15 (depends on 11)

The component of claim 11, wherein the endoscopic case data further comprises one or more of:
case identifying data;
external data received at the endoscopic system from a source external to the endoscopic system; and
input from a user.

### Claim 16 (depends on 11)

The component of claim 11, wherein the data capture engine captures endoscopic case data at intervals during the endoscopic procedure by one or more of flagging, tagging, organizing, and structuring image analysis data to correlate endoscopic case data for a point in time.

### Claim 17 (depends on 11)

The component of claim 11, wherein the endoscopic procedure analysis engine analyzes the set of data using one or more algorithms.

### Claim 18 (depends on 11)

The component of claim 11, wherein the state data comprises one or more of:
a state of a setting for a component of the one or more components of the endoscopic system;
a state of a parameter for a component of the one or more components of the endoscopic system;
any change made by the control engine to one or more of a setting and a parameter for a component of the one or more components of the endoscopic system; and
any change made by a user to one or more of a setting and a parameter for a component of the one or more components of the endoscopic system.

### Claim 19 (depends on 11)

The component of claim 11, wherein the start time comprises a case start time, and the end time comprises a case end time, each of which is derived based on an out of body characteristic corresponding to image data of a view from outside the body of a patient provided by an endoscope of the endoscopic system.

### Claim 20 (depends on 11)

The component of claim 11, wherein the endoscopic procedure analysis engine derives additional data regarding average pump pressure in the endoscopic case.