# US 2021/0275145 A1 · Augmented reality interface for assisting a user to operate an ultrasound device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0275145 A1
- Document ID: US20210275145A1
- Status: Published application
- Kind: A1
- Application number: 17/330,343
- Filed: 2021-05-25
- Published: 2021-09-09
- Earliest claimed: 2016-06-20
- Assignees: Butterfly Network, Inc.
- Applicants: Butterfly Network, Inc.
- Inventors: Matthew de Jonge; Robert Schneider; David Elgena; Alex Rothberg; Jonathan M. Rothberg; Michal Sofka; Tomer Gafner; Karl Thiele; Abraham Neben
- Web page: https://patentgrep.com/patent/US20210275145A1

## CPC

- G06V 10/82: using neural networks
- A61B 8/02: Measuring pulse or heart rate
- A61B 8/06: Measuring blood flow
- A61B 8/065: {to determine blood output from the heart}
- A61B 8/085: {for locating body or organic structures, e.g. tumours, calculi, blood vessels, nodules}
- A61B 8/4427: {Device being portable or laptop-like}
- A61B 8/46: {Ultrasonic, sonic or infrasonic diagnostic devices with special arrangements for interfacing with the operator or the patient}
- A61B 8/469: {for selection of a region of interest}
- A61B 8/52: {Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves}
- A61B 8/5207: {involving processing of raw data to produce diagnostic data, e.g. for generating an image}
- A61B 8/5223: {for extracting a diagnostic or physiological parameter from medical diagnostic data (for algorithms to analyse biomedical images G06T7/0012)}
- A61B 8/585: {Automatic set-up of the device}
- G06F 18/24133: {Distances to prototypes}
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/0014: {using an image reference approach}
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 11/60: Creating or editing images; Combining images with text
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 30/19173: {Classification techniques}
- G06V 30/194: References adjustable by an adaptive method, e.g. learning
- G06V 40/67: {by interactive indications to the user}
- G16H 30/40: for processing medical images, e.g. editing
- G16H 50/30: for calculating health indices; for individual health risk assessment
- A61B 8/0833: {involving detecting or locating foreign bodies or organic structures}
- A61B 8/0883: {for diagnosis of the heart}
- A61B 8/4263: {using sensors not mounted on the probe, e.g. mounted on an external reference frame}
- A61B 8/463: {characterised by displaying multiple images or images and diagnostic data on one display}
- A61B 8/5215: {involving processing of medical diagnostic data}
- A61B 2034/2065: {Tracking using image or pattern recognition}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/378: {using ultrasound}
- A61B 2090/3937: {Visible markers}
- G06T 2207/10132: Ultrasound image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30048: Heart; Cardiac
- G06T 2207/30061: Lung
- G06T 2210/41: Medical
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

Aspects of the technology described herein relate to techniques for guiding an operator to use an ultrasound device. Thereby, operators with little or no experience operating ultrasound devices may capture medically relevant ultrasound images and/or interpret the contents of the obtained ultrasound images. For example, some of the techniques disclosed herein may be used to identify a particular anatomical view of a subject to image with an ultrasound device, guide an operator of the ultrasound device to capture an ultrasound image of the subject that contains the particular anatomical view, and/or analyze the captured ultrasound image to identify medical information about the subject.

## Claims

### Claim 1

A method, comprising:
capturing, using a camera coupled to a mobile device, an image of an ultrasound device being used by an operator when the ultrasound device is in the camera's field of view;
identifying a pose of the ultrasound device in the image of the ultrasound device;
generating, using at least one processor of the mobile device, a composite image at least in part by overlaying, onto the image of the ultrasound device and using the pose of the ultrasound device, an indication of a direction in which the operator is to move the ultrasound device;
presenting the composite image to the operator via a display of the mobile device; and
prior to generating the composite image, using the mobile device to:
analyze an ultrasound image captured by the ultrasound device by using a trained model, and
determine the indication using an output of the trained model.

### Claim 2 (depends on 1)

The method of claim 1, wherein the ultrasound device has a marker disposed thereon, and wherein capturing the image of the ultrasound device comprises obtaining an image of the marker.

### Claim 3 (depends on 2)

The method of claim 2, wherein identifying the pose of the ultrasound device comprises identifying a location of the marker in the image of the ultrasound device.

### Claim 4 (depends on 1)

The method of claim 1, further comprising obtaining the ultrasound image captured by the ultrasound device.

### Claim 5 (depends on 4)

The method of claim 4, further comprising presenting the ultrasound image with the composite image via the display of the mobile device.

### Claim 6 (depends on 1)

The method of claim 1, wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to rotate the ultrasound device.

### Claim 7 (depends on 1)

The method of claim 1, wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to reposition the ultrasound device.

### Claim 8 (depends on 1)

The method of claim 1, wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to move the ultrasound device up, down, left, or right.

### Claim 9

A system, comprising:
a mobile device comprising a display and at least one processor configured to:
capture, using a camera coupled to the mobile device, an image of an ultrasound device being used by an operator when the ultrasound device is in the camera's field of view;
identify a pose of the ultrasound device in the image of the ultrasound device;
generate, using the at least one processor of the mobile device, a composite image at least in part by overlaying, onto the image of the ultrasound device and using the pose of the ultrasound device, an indication of a direction in which the operator is to move the ultrasound device;
cause the display to present the composite image to the operator; and
prior to generating the composite image:
analyze an ultrasound image captured by the ultrasound device by using a trained model, and
determine the indication using an output of the trained model.

### Claim 10 (depends on 9)

The system of claim 9, wherein the ultrasound device comprises a marker disposed thereon, wherein the image of the ultrasound device comprises an image of the marker, and wherein the at least one processor is configured to identify the pose of the ultrasound device at least in part by identifying a location of the marker in the image of the ultrasound device.

### Claim 11 (depends on 10)

The system of claim 10, wherein the marker is selected from the group consisting of: a holographic marker, a dispersive marker, and an ArUco marker.

### Claim 12 (depends on 9)

The system of claim 9, further comprising the ultrasound device.

### Claim 13 (depends on 9)

The system of claim 9, wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to rotate the ultrasound device.

### Claim 14 (depends on 9)

The system of claim 9, wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to reposition the ultrasound device.

### Claim 15 (depends on 9)

The system of claim 9, wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to move the ultrasound device up, down, left, or right.

### Claim 16

At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one processor of a mobile device, cause the at least one processor to:
capture, using a camera coupled to the mobile device, an image of an ultrasound device being used by an operator when the ultrasound device is in the camera's field of view;
identify a pose of the ultrasound device in the image of the ultrasound device;
generate, using the at least one processor of the mobile device, a composite image at least in part by overlaying, onto the image of the ultrasound device and using the pose of the ultrasound device, an indication of a direction in which the operator is to move the ultrasound device;
cause the display to present the composite image to the operator; and
prior to generating the composite image:
analyze an ultrasound image captured by the ultrasound device by using a trained model, and
determine the indication using an output of the trained model.

### Claim 17 (depends on 16)

The at least one non-transitory computer-readable storage medium of claim 16, wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to rotate the ultrasound device.

### Claim 18 (depends on 16)

The at least one non-transitory computer-readable storage medium of claim 16, wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to reposition the ultrasound device.

### Claim 19 (depends on 16)

The at least one non-transitory computer-readable storage medium of claim 16, wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to move the ultrasound device up, down, left, or right.
