# US 12,001,939 B2 · Artificial intelligence (AI)-based guidance for an ultrasound device to improve capture of echo image views

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,001,939 B2
- Document ID: US12001939B2
- Status: Granted
- Kind: B2
- Application number: 17/219,670
- Filed: 2021-03-31
- Published: 2021-08-26
- Granted: 2024-06-04
- Earliest claimed: 2018-12-11
- Assignees: Eko.Ai Pte. Ltd.
- Applicants: Eko.Ai Pte. Ltd.
- Inventors: James Otis Hare, II; Su Ping Carolyn Lam; Yoran Hummel; Mathias Iversen; Andrie Ochtman
- Web page: https://patentgrep.com/patent/US12001939B2

## CPC

- G06N 3/045: Combinations of networks
- A61B 8/0883: {for diagnosis of the heart}
- A61B 8/14: Echotomography
- A61B 8/4245: {involving determining the position of the probe, e.g. with respect to an external reference frame or to the patient}
- A61B 8/463: {characterised by displaying multiple images or images and diagnostic data on one display}
- A61B 8/5292: {using additional data, e.g. patient information, image labeling, acquisition parameters}
- A61B 8/565: {involving data transmission via a network}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/063: using electronic means
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06N 3/098: Distributed learning, e.g. federated learning
- G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS
- G16H 30/40: for processing medical images, e.g. editing
- G16H 40/63: for local operation
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- G16H 70/20: relating to practices or guidelines
- A61B 8/488: {involving Doppler signals}
- G06N 3/047: Probabilistic or stochastic networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

Artificial intelligence (AI)-based guidance for an ultrasound device to improve capture of echo image views comprises receiving a plurality of the echo images captured by a probe of the ultrasound device, the ultrasound device including a display and a user interface (UI) that displays the echo images to a user. One or more neural networks process the echo images to continuously attempt to automatically classify the echo images by view type and generates corresponding classification confidence scores. The view type of the echo images are simultaneously displayed in the UI along with the echo images. Feedback indications are displayed in the UI of the ultrasound device to the user, where the feedback indications include which directions to move the probe of the ultrasound device so the probe can be placed in a correct position to capture and successfully classify the echo images.

## Claims

### Claim 1

A computer-implemented method for artificial intelligence (AI)-based guidance for an ultrasound device to improve capture of echo image views, the method comprising:
receiving, by at least one processor, a plurality of the echo images captured by a probe of the ultrasound device, the ultrasound device including a display and a user interface (UI) that displays the echo images to a user, the echo images comprising 2D images and Doppler modality images of a heart;
processing, by one or more neural networks, the echo images to continuously attempt to automatically classify the echo images by view type and generate corresponding classification confidence scores;
simultaneously displaying the view type of the echo images in the UI of the ultrasound device along with the echo images; and
displaying in the UI, feedback indications to the user including which directions to move the probe of the ultrasound device so the probe can be placed in a correct position to capture and successfully classify the echo images, the feedback indications comprising both informational guidance and directional guidance instructions,
the informational guidance comprising setup instructions, a list of view types to be captured, the view type currently being captured, confidence scores of the view types, remaining view types to be captured, all view types captured, or any combination thereof, and
the directional guidance instructions comprising i) macro-guidance instructions that instruct the user to adjust alignment or position of the probe to a known view type, or ii) micro-guidance instructions that instruct the user to optimize the view type of an echo image in progress, wherein the feedback indications comprise any combination of alphanumeric characters and graphical objects, audio, and any combination thereof.

### Claim 2 (depends on 1)

The method of claim 1, wherein the macro-guidance instructions and the micro-guidance instructions comprise probe movement instructions including: up/down or side-side, diagonal movements, rotation clockwise or counter-clockwise, tilt up/down or side-side, hold, or a combination thereof.

### Claim 3 (depends on 1)

The method of claim 1, further comprising associating the directional guidance instructions with a distance value or a time value.

### Claim 4 (depends on 1)

The method of claim 1, wherein displaying in the UI of the ultrasound device feedback indications to the user further comprises responsive to determining that the classification confidence scores meet a predetermined threshold, displaying the view type and a directional guidance instruction to the user to optimize the view type.

### Claim 5 (depends on 4)

The method of claim 4, further comprising responsive to determining that the classification confidence scores do not meet the predetermined threshold, displaying an estimated view type and a feedback instruction in a form of directional guidance instructions to adjust alignment or position of the probe to a known view type.

### Claim 6 (depends on 1)

The method of claim 1, further comprising displaying the feedback indications in a form of informational setup instructions in the UI of the ultrasound device prior to receiving the echo images.

### Claim 7 (depends on 1)

The method of claim 1, further comprising executing by the at least one processor an echo workflow engine, the at least one processor in electronic communication with the ultrasound device.

### Claim 8 (depends on 1)

The method of claim 1, further comprising executing by one or more servers an echo workflow engine in communication over a network with a client component executing on a computer, the computer in electronic communication with the ultrasound device.

### Claim 9 (depends on 1)

The method of claim 1, further comprising adding software as an echo workflow engine to the ultrasound device.

### Claim 10

A system, comprising:
a memory storing an artificial intelligence (AI) workflow instructions; and
one or more processors coupled to the memory, wherein when the artificial intelligence workflow instructions are executed cause the one or more processors to:
receive a plurality of echo images captured by a probe of an ultrasound device, the ultrasound device including a display and a user interface (UI) that displays the echo images to a user, the echo images comprising 2D images and Doppler modality images of a heart;
process, by one or more neural networks, the echo images to continuously attempt to automatically classify the echo images by view type and generate corresponding classification confidence scores;
simultaneously display the view type of the echo images in the UI of the ultrasound device along with the echo images; and
display in the UI, feedback indications to the user including which directions to move the probe of the ultrasound device so the probe can be placed in a correct position to capture and successfully classify the echo images, the feedback indications comprising both informational guidance and directional guidance instructions,
the informational guidance comprising setup instructions, a list of view types to be captured, the view type currently being captured, confidence scores of the view types, remaining view types to be captured, all view types captured, or any combination thereof, and
directional guidance instructions comprising i) macro-guidance instructions that instruct the user to adjust alignment or position of the probe to a known view type, or ii) micro-guidance instructions that instruct the user to optimize the view type of an echo image in progress; wherein the feedback indications comprise any combination of alphanumeric characters and graphical objects, audio, and any combination thereof.

### Claim 11 (depends on 10)

The system of claim 10, wherein the macro-guidance instructions and the micro-guidance instructions comprise probe movement instructions including: up/down or side-side, diagonal movements, rotation clockwise or counter-clockwise, tilt up/down or side-side, hold, or a combination thereof.

### Claim 12 (depends on 10)

The system of claim 10, further comprising associating the directional guidance instructions with a distance value or a time value.

### Claim 13 (depends on 10)

The system of claim 10, wherein displaying in the UI of the ultrasound device feedback indications to the user further comprises responsive to determining that the classification confidence scores meet a predetermined threshold, displaying the view type and a directional guidance instruction to the user to optimize the view type.

### Claim 14 (depends on 13)

The system of claim 13, further comprising responsive to determining that the classification confidence scores do not meet the predetermined threshold, displaying an estimated view type and a feedback instruction in a form of directional guidance instructions to adjust alignment or position of the probe to a known view type.

### Claim 15 (depends on 10)

The system of claim 10, further comprising displaying the feedback indications in a form of informational setup instructions in the UI of the ultrasound device prior to receiving the echo images.

### Claim 16 (depends on 10)

The system of claim 10, further wherein the AI workflow instructions execute on a computer in electronic communication with the ultrasound device.

### Claim 17 (depends on 10)

The system of claim 10, wherein the artificial intelligence workflow instructions execute on one or more servers in communication over a network with a client component executing on a computer, the computer in electronic communication with the ultrasound device.

### Claim 18 (depends on 10)

The system of claim 10, wherein the artificial intelligence workflow instructions are added to the ultrasound device.

### Claim 19

A non-transitory computer-readable medium containing program instructions for implementing artificial intelligence (AI)-based guidance for an ultrasound device to improve capture of echo image views, which when executed by a processor system configure the processor system for:
receiving a plurality of the echo images captured by a probe of the ultrasound device, the ultrasound device including a display and a user interface (UI) that displays the echo images to a user, the echo images comprising 2D images and Doppler modality images of a heart;
processing, by one or more neural networks, the echo images to continuously attempt to automatically classify the echo images by view type and generate corresponding classification confidence scores;
simultaneously displaying the view type of the echo images in the UI of the ultrasound device along with the echo images; and
displaying in the UI, feedback indications to the user including which directions to move the probe of the ultrasound device so the probe can be placed in a correct position to capture and successfully classify the echo images, the feedback indications comprising both informational guidance and directional guidance instructions,
the informational guidance comprising setup instructions, a list of view types to be captured, the view type currently being captured, confidence scores of the view types, remaining view types to be captured, all view types captured, or any combination thereof, and
directional guidance instructions comprising i) macro-guidance instructions that instruct the user to adjust alignment or position of the probe to a known view type, or ii) micro-guidance instructions that instruct the user to optimize the view type of an echo image in progress; wherein the feedback indications comprise any combination of alphanumeric characters and graphical objects, audio, and any combination thereof.