# US 2023/0267699 A1 · Methods and apparatuses for tele-medicine

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0267699 A1
- Document ID: US20230267699A1
- Status: Published application
- Kind: A1
- Application number: 18/137,049
- Filed: 2023-04-20
- Published: 2023-08-24
- Earliest claimed: 2019-01-07
- Assignees: Bfly Operations, Inc.
- Applicants: Bfly Operations, Inc.
- Inventors: Maxim Zaslavsky; Matthew de Jonge; David Elgena; Patrick Temple; Jason Quense; Aditya Ayyakad
- Web page: https://patentgrep.com/patent/US20230267699A1

## CPC

- G06V 10/17: {using hand-held instruments}
- A61B 8/463: {characterised by displaying multiple images or images and diagnostic data on one display}
- A61B 8/465: {adapted to display user selection data, e.g. icons or menus}
- A61B 8/468: {allowing annotation or message recording}
- A61B 8/469: {for selection of a region of interest}
- A61B 8/5223: {for extracting a diagnostic or physiological parameter from medical diagnostic data (for algorithms to analyse biomedical images G06T7/0012)}
- A61B 8/54: {Control of the diagnostic device}
- A61B 8/565: {involving data transmission via a network}
- A61B 8/582: {Remote testing of the device}
- G06T 19/003: {Navigation within 3D models or images}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G09B 5/02: with visual presentation of the material to be studied, e.g. using film strip
- G09B 5/14: with provision for individual teacher-student communication
- G09B 23/286: {for scanning or photography techniques, e.g. X-rays, ultrasonics}
- G16H 40/67: for remote operation
- G06T 2219/2016: Rotation, translation, scaling
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

Aspects of the technology described herein relate to operator processing devices and instructor processing device for tele-medicine. The instructor processing device may be configured to receive, from an instruction interface, a selection of an instruction for moving an ultrasound device. The operator processing device may be configured to determine a pose of the ultrasound device relative to the operator processing device. The instructor processing device and the operator processing device may be configured to display in an operator video, based on the pose of the ultrasound device relative to the operator processing device and based on the selected instruction, a directional indicator for moving the ultrasound device. The instructor processing device may also be configured to display, based on the pose of the ultrasound device relative to the operator processing device, orientation indicators in the instruction interface and/or the operator video.

## Claims

### Claim 1

An ultrasound system, comprising:
an instructor processing device configured to:
receive a current pose of an ultrasound device relative to an operator processing device from the operator processing device; and
display, in an operator video displayed on the instructor processing device, based on the current pose of the ultrasound device relative to the operator processing device, an orientation indicator indicating the current pose of the ultrasound device relative to the operator processing device.

### Claim 2 (depends on 1)

The ultrasound system of claim 1, wherein the operator video depicts the ultrasound device.

### Claim 3 (depends on 1)

The ultrasound system of claim 1, wherein the orientation indicator is displayed in the operator video such that the orientation indicator appears to be a part of a real-world environment in the operator video.

### Claim 4 (depends on 1)

The ultrasound system of claim 1, wherein the instructor processing device is configured, when displaying the orientation indicator indicating the current pose of the ultrasound device relative to the operator processing device, to:
determine a default position and orientation of the orientation indicator in three-dimensional space for a default pose of the ultrasound device relative to the operator processing device;
position and/or orient the orientation indicator in three-dimensional space from the default position and orientation based on a difference between the current pose and the default pose of the ultrasound device relative to the operator processing device; and
project the orientation indicator from its three-dimensional position and orientation into two-dimensional space for display in the operator video displayed on the instruction processing device.

### Claim 5 (depends on 1)

The ultrasound system of claim 1, wherein the orientation indicator indicates an orientation of a marker on the ultrasound device relative to the operator processing device.

### Claim 6 (depends on 5)

The ultrasound system of claim 5, wherein:
the orientation indicator comprises a ring and a ball;
the ring is centered in the operator video approximately at a tail of the ultrasound device and orientated approximately within a plane orthogonal to a longitudinal axis of the ultrasound device; and
the ball is located in the operator video on the ring such that a line from the ring to the marker on the ultrasound device is approximately parallel to the longitudinal axis of the ultrasound device.

### Claim 7

An ultrasound system, comprising:
an instructor processing device configured to:
receive a current pose of an ultrasound device relative to an operator processing device from the operator processing device; and
display, in an instruction interface displayed on a display screen of the instructor processing device, based on the pose of the ultrasound device relative to the operator processing device, an orientation indicator indicating the current pose of the ultrasound device relative to the operator processing device.

### Claim 8 (depends on 7)

The ultrasound system of claim 7, wherein the instructor processing device is configured, when displaying the orientation indicator indicating the current pose of the ultrasound device relative to the operator processing device, to:
determine, based on the current pose of the ultrasound device relative to the operator processing device, two points in three-dimensional space along an axis of the ultrasound device;
project the two points in three-dimensional space into two two-dimensional points in an operator video captured by the operator processing device; and
display the orientation indicator at an angle relative to a horizontal axis of the display screen of the instructor processing device that is equivalent to an angle between a line formed by the two two-dimensional points and the horizontal axis of the display screen of the instructor processing device.

### Claim 9 (depends on 7)

The ultrasound system of claim 7, wherein the orientation indicator illustrates a direction a marker on the ultrasound device is pointing relative to the operator processing device.

### Claim 10 (depends on 7)

The ultrasound system of claim 7, wherein the orientation indicator illustrates two-dimensionally a three-dimensional pose of a marker on the ultrasound device.

### Claim 11 (depends on 7)

The ultrasound system of claim 7, wherein the instruction interface comprises a circle, and the orientation indicator is at a particular position around the circle that indicates the current pose of the ultrasound device relative to the operator processing device.

### Claim 12

An ultrasound system, comprising:
an instructor processing device configured to:
display a translation instruction interface for selecting an instruction to translate an ultrasound device, the translation instruction interface comprising an arrow rotatable by an instructor.

### Claim 13 (depends on 12)

The ultrasound system of claim 12, wherein the instructor processing device is further configured to:
receive a selection of an instruction to translate the ultrasound device from the translation instruction interface based on an angle of the arrow rotatable by the instructor.

### Claim 14 (depends on 12)

The ultrasound system of claim 12, wherein the translation instruction interface includes an orientation indicator indicating a pose of the ultrasound device relative to an operator processing device.

### Claim 15 (depends on 14)

The ultrasound system of claim 14, wherein the orientation indicator illustrates a direction a marker on the ultrasound device is pointing relative to the operator processing device.

### Claim 16 (depends on 14)

The ultrasound system of claim 14, wherein the orientation indicator illustrates two-dimensionally a three-dimensional pose of a marker on the ultrasound device.

### Claim 17 (depends on 12)

The ultrasound system of claim 12, wherein the instruction to translate the ultrasound device is based on an angle formed by the arrow rotatable by the instructor.

### Claim 18 (depends on 12)

The ultrasound system of claim 12, further comprising:
an operator processing device configured to:
display, in an operator video captured by the operator processing device and displayed on an operator processing device, a two-dimensional arrow indicating a direction for translating the ultrasound device, wherein displaying the arrow comprises:
determine, based on a pose of the ultrasound device relative to the operator processing device, two points in three-dimensional space along an axis of the ultrasound device;
project the two points in three-dimensional space into two two-dimensional points in the operator video;
calculate an angle between a line formed by the two points and a horizontal axis of the operator video;
subtract the angle from a desired instruction angle to produce a final angle, wherein the desired instruction angle is determined from the arrow rotatable by the instructor on the translation instruction interface displayed on the instructor processing device;
determine, based on the pose of the ultrasound device relative to the operator processing device, an arrow in three-dimensional space pointing along the final angle; and
project the arrow in three-dimensional space into the two-dimensional arrow displayed in the operator video and indicating the direction for translating the ultrasound device.

### Claim 19 (depends on 12)

The ultrasound system of claim 12, wherein the instructor processing device is further configured to display:
a rotation instruction interface for selecting an instruction to rotate the ultrasound device; and
a tilt instruction interface for selecting an instruction to tilt the ultrasound device.

### Claim 20 (depends on 12)

The ultrasound system of claim 12, wherein the instructor processing device is further configured to receive a manually drawing from the instructor on an ultrasound image displayed by the instructor processing device.
