# US 2021/0059765 A1 · Instrument image reliability systems and methods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0059765 A1
- Document ID: US20210059765A1
- Status: Published application
- Kind: A1
- Application number: 17/005,165
- Filed: 2020-08-27
- Published: 2021-03-04
- Earliest claimed: 2019-08-30
- Assignees: Auris Health, Inc.
- Applicants: Auris Health, Inc.
- Inventors: Menglong Ye
- Web page: https://patentgrep.com/patent/US20210059765A1

## CPC

- A61B 5/066: {Superposing sensor position on an image of the patient, e.g. obtained by ultrasound or x-ray imaging}
- A61B 1/000096: {using artificial intelligence}
- A61B 1/0002: {provided with data storages}
- A61B 1/00149: {using articulated arms}
- A61B 1/05: characterised by the image sensor, e.g. camera, being in the distal end portion
- A61B 1/2676: {Bronchoscopes}
- A61B 5/0036: {including treatment, e.g., using an implantable medical device, ablating, ventilating}
- A61B 5/7221: {Determining signal validity, reliability or quality (preventing, reducing or removing noise induced by motion artefacts A61B5/7207; noise originating from a therapeutic or surgical apparatus A61B5/7217)}
- A61B 5/7267: {involving training the classification device}
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 34/37: Leader-follower robots (A61B34/35 takes precedence)
- A61B 34/74: {Manipulators with manual electric input means}
- A61B 90/361: {Image-producing devices, e.g. surgical cameras}
- A61B 90/50: Supports for surgical instruments, e.g. articulated arms
- A61G 12/001: {Trolleys for transport of medicaments, food, linen, nursing supplies (carts specially adapted for surgery or diagnosis A61B50/13; hand carts B62B3/00)}
- A61G 13/0018: {Physician's examining tables}
- A61G 13/04: tiltable around transverse or longitudinal axis
- A61G 13/1235: {Arms}
- A61G 13/125: {Ankles or feet}
- A61G 13/1285: {having modular surface parts, e.g. being replaceable or turnable}
- A61G 13/129: {having surface parts for adaptation of the size, e.g. for extension or reduction (tables with different adjustable sections A61G13/08)}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2433: Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/73: using feature-based methods
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- A61B 10/0233: {Pointed or sharp biopsy instruments}
- A61B 46/10: specially adapted for instruments{, e.g. microscopes}
- A61B 50/13: Trolleys{, e.g. carts}
- A61B 90/30: Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure
- A61B 2017/00477: {Coupling (A61B2017/0046 takes precedence)}
- A61B 2017/00809: {Lung operations}
- A61B 2034/105: {Modelling of the patient, e.g. for ligaments or bones}
- A61B 2034/2051: {Electromagnetic tracking systems}
- A61B 2034/2059: {Mechanical position encoders}
- A61B 2034/2061: {using shape-sensors, e.g. fiber shape sensors with Bragg gratings}
- A61B 2034/2065: {Tracking using image or pattern recognition}
- A61B 2034/301: {for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes}
- A61B 2090/376: {using X-rays, e.g. fluoroscopy}
- A61B 2217/005: with suction drainage system
- A61B 2217/007: with irrigation system
- A61B 2505/05: Surgical care
- A61B 2576/00: Medical imaging apparatus involving image processing or analysis (A61B1/00009, A61B6/52 and A61B8/52 take precedence)
- A61G 2210/50: for radiography
- G06T 2207/10068: Endoscopic image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30061: Lung
- G06T 2207/30168: Image quality inspection
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- A61G: TRANSPORT, PERSONAL CONVEYANCES OR ACCOMMODATION SPECIALLY ADAPTED FOR PATIENTS OR DISABLED PERSONS (appliances for aiding patients or disabled persons to walk A61H3/00); OPERATING TABLES OR CHAIRS; CHAIRS FOR DENTISTRY; FUNERAL DEVICES (embalming corpses A01N1/00)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Described herein are systems and methods related to image management. A system may include an instrument that includes an elongate body and an imaging device. The elongate body can be configured to be inserted into a luminal network. The imaging device may be positioned at a distal tip of the elongate body. The system may receive from the imaging device one or more images captured when the elongate body is within the luminal network. For each of the images, the system may determine one or more metrics that are indicative of a reliability of an image for localization of the distal tip of the elongate body within the luminal network. The system may determine a reliability threshold value for each of the one or more metrics. The system can utilize the one or more images based on whether the one or more metrics meet corresponding reliability threshold values.

## Claims

### Claim 1

A system comprising:
an instrument comprising:
an elongate body configured to be inserted into a luminal network, and
an imaging device positioned at a distal tip of the elongate body;
at least one computer-readable memory having stored thereon executable instructions; and
one or more processors in communication with the at least one computer-readable memory and configured to execute the instructions to cause the system to at least:
receive, from the imaging device, one or more images captured when the elongate body is within the luminal network;
for each of the one or more images, determine one or more metrics indicative of a reliability of an image for localization of the distal tip of the elongate body within the luminal network;
determine a reliability threshold value for each of the one or more metrics; and
in response to each of the one or more metrics for the image meeting a corresponding reliability threshold value, utilize the image for the localization of the distal tip.

### Claim 2 (depends on 1)

The system of claim 1, wherein the one or more processors are configured to execute the instructions to cause the system to at least:
access a machine learning model of images meeting one or more reliability threshold values; and
determine, based on the machine learning model, whether the one or more metrics for each of the one or more captured images meet the corresponding reliability threshold values.

### Claim 3 (depends on 2)

The system of claim 2, wherein the machine learning model comprises a convolutional neural network trained to differentiate between (i) images with one or more metrics that meet the corresponding reliability threshold values and (ii) images with one or more metrics that do not meet the corresponding reliability threshold values.

### Claim 4 (depends on 1)

The system of claim 1, wherein the one or more metrics comprise at least one of:
a first metric indicative of a level of blur in the image;
a second metric indicative of an amount of specular highlights in the image; and
a third metric indicative of an availability of visual features in the image.

### Claim 5 (depends on 4)

The system of claim 4, wherein the utilizing of the image comprises:
assigning a first weighting parameter to the first metric;
assigning a second weighting parameter to the second metric;
assigning a third weighting parameter to the third metric; and
determining a composite metric based on a linear combination of the first, second, and third metrics with the respective first, second, and third weighting parameters, wherein the composite metric is indicative of the reliability of the image for the localization of the distal tip.

### Claim 6 (depends on 4)

The system of claim 4, wherein determining the one or more metrics comprises:
identifying image values corresponding to two-dimensional coordinates of the image,
wherein each image value is based on color values at the corresponding coordinate.

### Claim 7 (depends on 6)

The system of claim 6, wherein determining the one or more metrics comprises determining the first metric based on:
performing a Laplacian operator on a modified image of the image to obtain a matrix; and
determining a variance of the matrix associated with the modified image.

### Claim 8 (depends on 6)

The system of claim 6, wherein determining the one or more metrics comprises determining the second metric based on:
determining saturation and lightness values for the image; and
determining a ratio of at least one of saturation or value to an area of the image.

### Claim 9 (depends on 8)

The system of claim 8, wherein determining the second metric comprises determining a variance of lightness values between a first point and a second point of the image, the first and second points being adjacent to each other.

### Claim 10 (depends on 8)

The system of claim 8, wherein determining the second metric comprises identifying a number of pixels that have a saturation value below a threshold saturation level.

### Claim 11 (depends on 6)

The system of claim 6, wherein determining the one or more metrics comprises determining the third metric based on:
using at least one of SIFT, SURF, BRIEF, BRISK, or FAST to identify a number of features in the image.

### Claim 12 (depends on 6)

The system of claim 6, wherein identifying a number of features comprises identifying at least one of an edge or a corner in the image.

### Claim 13 (depends on 1)

The system of claim 1, wherein the utilizing of the image comprises assigning a reliability factor to each of the one or more images, the reliability factor based on the one or more metrics and indicative of the reliability of the image for the localization of the distal tip.

### Claim 14 (depends on 1)

The system of claim 1, wherein the one or more processors are further configured to execute the instructions to cause the system to receive from a user each of the reliability threshold values corresponding to the one or more metrics.

### Claim 15 (depends on 1)

The system of claim 1, wherein the one or more processors are further configured to execute the instructions to cause the system to automatically set a first reliability threshold value higher than a second reliability threshold value based on corresponding first and second metrics of the one or more metrics.

### Claim 16 (depends on 1)

The system of claim 1, wherein the one or more processors are further configured to execute the instructions to cause the system to access preoperative model data comprising virtual images.

### Claim 17 (depends on 16)

The system of claim 16, wherein accessing preoperative model data comprises comparing the one or more images captured when the elongate body is within the luminal network with the virtual images.

### Claim 18 (depends on 17)

The system of claim 17, wherein the one or more processors are further configured to execute the instructions to cause the system to determine a location of the elongate body within the luminal network based on comparing the one or more images captured when the elongate body is within the luminal network with the virtual images.

### Claim 19

A method comprising:
using an image device disposed at a distal end of an elongate instrument, receiving one or more images captured when the elongate instrument is within a luminal network;
accessing a machine learning model trained on images meeting one or more threshold quality metrics; and
determining, based on the machine learning model, whether the one or more images meet the one or more threshold quality metrics.

### Claim 20 (depends on 19)

The method of claim 19, further comprising determining, based on the machine learning model, one or more quality metrics indicative of a reliability of an image for localization of the distal end of the elongate instrument within the luminal network.

### Claim 21 (depends on 19)

The method of claim 19, further comprising determining the one or more threshold quality metrics.

### Claim 22 (depends on 19)

The method of claim 19, further comprising filtering the image from being used for the localization of the distal end in response to a determination that at least one of the one or more quality metrics of the image fail to meet the corresponding one or more threshold quality metrics.

### Claim 23 (depends on 19)

The method of claim 19, further comprising determining a location of the elongate instrument within the luminal network.

### Claim 24 (depends on 23)

The method of claim 23, wherein determining the location of the elongate instrument within the luminal network comprises comparing the one or more captured images with preoperative images of a luminal network.

### Claim 25 (depends on 24)

The method of claim 24, wherein the preoperative images comprise computer-generated images.

### Claim 26 (depends on 24)

The method of claim 24, wherein the preoperative images comprise images obtained from a computerized tomography (CT) scan.

### Claim 27

A system comprising:
an instrument comprising:
an elongate body configured to be inserted into a luminal network, and
an imaging device positioned at a distal tip of the elongate body;
at least one computer-readable memory having stored thereon executable instructions; and
one or more processors in communication with the at least one computer-readable memory and configured to execute the instructions to cause the system to at least:
receive, from the imaging device, one or more images captured when the elongate body is within the luminal network;
for each of the one or more images, determine one or more metrics indicative of a reliability of an image for localization of the distal tip of the elongate body within the luminal network, wherein the one or more metrics comprise at least a first metric indicative of a level of blur in the image;
determine a reliability threshold value for each of the one or more metrics; and
in response to each of the one or more metrics for the image meeting a corresponding reliability threshold value, utilize the image for the localization of the distal tip.
