# US 2020/0286405 A1 · Method for developing a machine learning model of a neural network for classifying medical images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0286405 A1
- Document ID: US20200286405A1
- Status: Published application
- Kind: A1
- Application number: 16/727,844
- Filed: 2019-12-26
- Published: 2020-09-10
- Earliest claimed: 2017-01-24
- Assignees: Tienovix, LLC
- Applicants: Tienovix, LLC
- Inventors: William R. BURAS; Craig S. RUSSELL; Kyle Q. NGUYEN
- Web page: https://patentgrep.com/patent/US20200286405A1

## CPC

- G09B 23/286: {for scanning or photography techniques, e.g. X-rays, ultrasonics}
- A61B 8/06: Measuring blood flow
- A61B 8/085: {for locating body or organic structures, e.g. tumours, calculi, blood vessels, nodules}
- A61B 8/4245: {involving determining the position of the probe, e.g. with respect to an external reference frame or to the patient}
- A61B 8/4254: {using sensors mounted on the probe}
- A61B 8/4263: {using sensors not mounted on the probe, e.g. mounted on an external reference frame}
- A61B 8/466: {adapted to display 3D data}
- A61B 8/467: {characterised by special input means}
- A61B 8/52: {Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves}
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 90/361: {Image-producing devices, e.g. surgical cameras}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06N 20/00: Machine learning
- G09B 5/065: {Combinations of audio and video presentations, e.g. videotapes, videodiscs, television systems}
- G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS
- G16H 40/63: for local operation
- A61B 2034/2048: {using an accelerometer or inertia sensor}
- A61B 2034/2055: {Optical tracking systems}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/378: {using ultrasound}
- A61B 2090/3945: {Active visible markers, e.g. light emitting diodes}
- A61B 2562/0219: Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
- G02B 27/017: {Head mounted}
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS

## Abstract

Methods for developing a machine learning model of a neural network for classifying medical images.

## Claims

### Claim 1

A method for developing a machine learning model of a neural network for classifying images for a medical procedure using an ultrasound system, the method comprising:
A) performing a first medical procedure using an ultrasound system;
B) automatically capturing a plurality of ultrasound images during the performance of the first medical procedure, wherein each of the plurality of ultrasound images is captured at a defined sampling rate according to defined image capture criteria;
C) providing a plurality of feature modules, wherein each feature module defines a feature which may be present in an image captured during the medical procedure;
D) automatically analyzing each image using the plurality of feature modules;
E) automatically determining, for each image, whether or not each of the plurality of features is present in the image, based on the analysis of each imagine using the feature modules;
F) automatically labeling each image as belonging to one class of a plurality of image classes associated with the medical procedure;
G) automatically splitting the plurality of images into a training set of images and a validation set of images;
H) providing a deep machine learning (DML) platform having a neural network to be trained loaded thereon, the DML platform having a plurality of adjustable parameters for controlling the outcome of a training process;
I) feeding the training set of images into the DML platform;
J) performing the training process for the neural network to generate a machine learning model of the neural network;
K) obtaining training process metrics of the ability of the generated machine learning model to classify images during the training process, wherein the training process metrics comprise at least one of a loss metric, an accuracy metric, and an error metric for the training process;
L) determining whether each of the at least one training process metrics is within an acceptable threshold for the training process metric;
M) if one or more of the training process metrics are not within an acceptable threshold, adjusting one or more of the plurality of adjustable DML parameters and repeating steps J, K, and L using the training set of images;
N) if each of the training process metrics is within an acceptable threshold for the metric, performing a validation process using the validation set of images;
O) obtaining validation process metrics of the ability of the generated machine learning model to classify images during the validation process, wherein the validation process metrics comprise at least one of a loss metric, an accuracy metric, and an error metric for the validation process;
P) determining whether each of the validation process metrics is within an acceptable threshold for the validation process metric;
Q) if one or more of the validation process metrics are not within an acceptable threshold, adjusting one or more of the plurality of adjustable DML parameters and repeating steps J-P; and
R) if each of the validation process metrics is within an acceptable threshold for each metric, storing the machine learning model for the neural network.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
S) receiving, after storing the machine learning model for the neural network, a plurality of images from a user performing the first medical procedure using an ultrasound system;
T) using the stored machine learning model to classify each of the plurality of images received from the ultrasound system during the second medical procedure.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
S) using the stored machine learning model for the neural network to classify a plurality of ultrasound images for a user performing the first medical procedure.

### Claim 4 (depends on 1)

The method of claim 1, wherein performing the training process comprises iteratively computing weights and biases for each of the nodes of the neural network using feed-forward and back-propagation until the accuracy of the network in classifying images reaches an acceptable threshold level of accuracy.

### Claim 5 (depends on 1)

The method of claim 1, wherein performing the validation process comprises using the machine learning model generated by the training process to classify the images of the validation set of image data.

### Claim 6 (depends on 1)

The method of claim 1, further comprising stopping the method if steps J, K, and L have been repeated more than a threshold number of repetitions.

### Claim 7 (depends on 6)

The method of claim 6, further comprises stopping the method if steps N-Q have been repeated more than a threshold number of repetitions.

### Claim 8 (depends on 1)

The method of claim 1, wherein providing a deep machine learning (DML) platform comprises providing a DML platform having at least one adjustable parameter selected from learning rate constraints, number of epochs to train, epoch size, minibatch size, and momentum constraints.

### Claim 9 (depends on 8)

The method of claim 8, wherein adjusting one or more of the plurality of adjustable DML parameters comprises automatically adjusting said one or more parameters using a particle swarm optimization algorithm.

### Claim 10 (depends on 1)

The method of claim 1, wherein automatically splitting the plurality of images comprises automatically splitting the plurality of images into a training set comprising from 70% to 90% of the plurality of images, and a validation set comprising from 10% to 30% of the plurality of images.

### Claim 11 (depends on 1)

The method of claim 1, wherein automatically labeling each image further comprises isolating one or more of the features present in the image using a boundary indicator selected from a bounding box, a bounding circle, a bounding ellipse, and an irregular bounding region.

### Claim 12 (depends on 1)

The method of claim 1, wherein obtaining training process metrics comprises obtaining at least one of average cross-entropy error for all epochs and average classification error for all epochs.

### Claim 13 (depends on 1)

The method of claim 1, wherein determining whether each of the training process metrics are within an acceptable threshold comprises determining whether average cross-entropy error for all epochs is less than a threshold selected from 5% to 10%, and average classification error for all epochs is less than a threshold selected from 15% to 10%.

### Claim 14 (depends on 1)

The method of claim 1, wherein step A) is performed by an expert.

### Claim 15

A method for developing a machine learning model of a neural network for classifying images for a medical procedure using a medical imaging system, the method comprising:
A) performing a first medical procedure using an imaging system;
B) capturing a plurality of images during the performance of the first medical procedure;
C) providing a plurality of feature modules, each feature module defining a feature which may be present in an image captured during the medical procedure;
D) automatically analyzing each image using the plurality of feature modules;
E) automatically determining, for each image, whether or not each of the plurality of features is present in the image, based on the analysis of each imagine using the feature modules;
F) labeling each image as belonging to one class of a plurality of image classes associated with the medical procedure;
G) automatically splitting the plurality of images into a training set of images and a validation set of images;
H) providing a deep machine learning (DML) platform having a neural network to be trained loaded thereon, the DML platform having a plurality of adjustable parameters for controlling the outcome of a training process;
I) feeding the training set of images into the DML platform;
J) performing the training process for the neural network to generate a machine learning model of the neural network;
K) obtaining training process metrics of the ability of the generated machine learning model to classify images during the training process, wherein the training process metrics comprise at least one of a loss metric, an accuracy metric, and an error metric for the training process;
L) determining whether each of the at least one training process metrics is within an acceptable threshold for the training process metric;
M) if one or more of the training process metrics are not within an acceptable threshold, adjusting one or more of the plurality of adjustable DML parameters and repeating steps J, K, and L using the training set of images;
N) if each of the training process metrics is within an acceptable threshold for the metric, performing a validation process using the validation set of images;
O) obtaining validation process metrics of the ability of the generated machine learning model to classify images during the validation process, wherein the validation process metrics comprise at least one of a loss metric, an accuracy metric, and an error metric for the validation process;
P) determining whether each of the validation process metrics is within an acceptable threshold for the validation process metric;
Q) if one or more of the validation process metrics are not within an acceptable threshold, adjusting one or more of the plurality of adjustable DML parameters and repeating steps J-P; and
R) if each of the validation process metrics is within an acceptable threshold for each metric, storing the machine learning model for the neural network.

### Claim 16 (depends on 15)

The system of claim 15, wherein capturing a plurality of images during the performance of the first medical procedure comprises automatically capturing images at a defined sampling rate according to defined image capture criteria.

### Claim 17 (depends on 15)

The system of claim 15, further comprising:
S) using the stored machine learning model for the neural network to classify a plurality of images captured by a user of the imaging system during the performance of the first medical procedure.

### Claim 18 (depends on 15)

The method of claim 15, wherein performing the training process comprises iteratively computing weights and biases for each of the nodes of the neural network using feed-forward and back-propagation until the accuracy of the network in classifying images reaches an acceptable threshold level of accuracy.

### Claim 19 (depends on 15)

The method of claim 15, wherein the imaging system is an ultrasound imaging system.

### Claim 20 (depends on 15)

The method of claim 15, wherein labeling each image as belonging to one class of a plurality of image classes associated with the medical procedure comprises automatically labeling each image.
