# US 12,758,177 B2 · Augmented reality man-machine interface for robotic surgery

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,758,177 B2
- Document ID: US12758177B2
- Status: Granted
- Kind: B2
- Application number: 18/465,388
- Filed: 2023-09-12
- Published: 2024-03-14
- Granted: 2026-10-13
- Earliest claimed: 2021-03-10
- Assignees: OnPoint Medical, Inc.
- Applicants: OnPoint Medical, Inc.
- Inventors: Daniel Steines; Philipp K. Lang
- Web page: https://patentgrep.com/patent/US12758177B2

## CPC

- A61B 6/548: {Remote control of the apparatus or devices}
- A61B 6/469: {for selecting a region of interest [ROI]}
- A61B 6/5294: {involving using additional data, e.g. patient information, image labeling, acquisition parameters}
- A61B 90/37: {Surgical systems with images on a monitor during operation}
- G02B 27/0172: {characterised by optical features}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- G02B 2027/0134: {of stereoscopic type}
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0141: {characterised by the informative content of the display}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems, devices and methods for augmented reality guidance of surgical procedures using multiple head mounted displays or other augmented reality display devices are described. In addition, systems, devices and methods using head mounted displays or other augmented reality display systems for operating surgical robots and/or imaging systems are described.

## Claims

### Claim 1

A system, comprising:
at least one head mounted display,
at least one camera or scanning device, wherein the at least one camera or scanning device is configured to track real-time information of the at least one head mounted display, of a patient table used with an imaging system, and of the imaging system,
inertial measurement units (IMU), wherein at least a first IMU is integrated or attached to the at least one head mounted display, wherein at least a second IMU is integrated or attached to the patient table, wherein at least a third IMU is integrated or attached to the imaging system,
wherein the at least first IMU is configured for real time tracking of the at least one head mounted display, wherein the at least second IMU is configured for real time tracking of the patient table, wherein the at least third IMU is configured for real time tracking of the imaging system,
a first computing system comprising one or more computer processors,
wherein the first computing system is configured to obtain the real-time tracking information from the at least one camera or scanning device and the at least first, the at least second, and the at least third IMU,
wherein the first computing system is configured for wireless transmission of the real-time tracking information of the at least one head mounted display, the patient table, and the imaging system,
a second computing system comprising one or more computer processors,
wherein the second computing system is configured for wireless reception of the real-time tracking information of the at least one head mounted display, the patient table, and the imaging system,
wherein the second computing system is configured to generate a 3D stereoscopic view comprising a 3D representation of one or more components of the imaging system, wherein the 3D representation is fixed or anchored in relationship to the one or more components of the imaging system,
wherein the at least one head mounted display is configured to display the 3D stereoscopic view, and
wherein the 3D stereoscopic view is updated in real-time for movement of the at least one head mounted display, the patient table, the imaging system or a combination thereof.

### Claim 2 (depends on 1)

The system of claim 1, wherein the second computing system is integrated with the at least one head mounted display, or wherein the second computing system is separate from the at least one head mounted display and is connected to a display unit of the at least one head mounted display using at least one cable.

### Claim 3 (depends on 1)

The system of claim 1, wherein the wireless transmission, the wireless reception, or both comprise a WiFi signal, a LiFi signal, a Bluetooth signal or a combination thereof.

### Claim 4 (depends on 1)

The system of claim 1, wherein the camera or scanning device is separate from the at least one head mounted display or wherein the camera or scanning device is integrated or attached to the at least one head mounted display.

### Claim 5 (depends on 1)

The system of claim 1, wherein the wireless transmission comprises sending data packets comprising the real-time tracking information of the at least one head mounted display, the patient table, and the imaging system, at a rate of 20 Hz or greater.

### Claim 6 (depends on 1)

The system of claim 1, wherein the wireless reception comprises receiving data packets comprising the real-time tracking information of the at least one head mounted display, the patient table, and the imaging system, at a rate of 20 Hz or greater.

### Claim 7 (depends on 1)

The system of claim 1, wherein the real-time tracking information comprises one or more coordinates, and wherein the one or more coordinates comprise coordinates of the patient table, the imaging system, and the at least one head mounted display.

### Claim 8 (depends on 1)

The system of claim 1, wherein the imaging system comprises an x-ray system, a fluoroscopy system, a C-arm, a 3D C-arm, a digital tomosynthesis imaging system, an angiography system, a bi-planar angiography system, a 3D angiography system, a CT scanner, a PET scanner, a SPECT scanner, a nuclear scintigraphy system, or a combination thereof.

### Claim 9 (depends on 1)

The system of claim 1, wherein the stereoscopic view comprises a 3D representation of at least a portion of the patient table.

### Claim 10 (depends on 1)

The system of claim 1, wherein the at least one camera or scanning device is configured to track at least a portion of the patient and wherein the 3D stereoscopic view comprises a 3D representation of at least a portion of the patient.

### Claim 11 (depends on 1)

The system of claim 1, comprising at least a fourth IMU is applied to the patient, wherein the at least fourth IMU is configured for tracking the patient, and wherein the 3D stereoscopic view comprises a 3D representation of at least a portion of the patient.

### Claim 12 (depends on 1)

The system of claim 1, wherein the real-time tracking information from the at least one camera or scanning device and the IMUs is transmitted and received simultaneously.

### Claim 13 (depends on 1)

The system of claim 1, wherein the real-time tracking information from the at least one camera or scanning device and the IMUs is transmitted and received separately.

### Claim 14 (depends on 1)

The system of claim 1, wherein the first computing system, the second computing system, or the first and the second computing system comprise a collision detection module.

### Claim 15 (depends on 14)

The system of claim 14, wherein the collision detection module is configured for detecting multi-body collision.

### Claim 16 (depends on 15)

The system of claim 15, wherein the collision detection module is configured to detect a collision and change the stereoscopic view responsive to the collision.

### Claim 17 (depends on 16)

The system of claim 16, wherein the change of the stereoscopic view comprises a color change.

### Claim 18 (depends on 14)

The system of claim 14, wherein the collision detection module is configured to utilize the real-time tracking information.

### Claim 19 (depends on 18)

The system of claim 18, wherein the first computing system, the second computing system, or the first and the second computing system comprising the collision detection module are configured to generate one or more event messages.

### Claim 20 (depends on 19)

The system of claim 19, wherein the one or more event messages comprise information about a position, orientation or combination thereof of the imaging system.