# US 11,992,373 B2 · Augmented reality headset with varied opacity for navigated robotic surgery

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,992,373 B2
- Document ID: US11992373B2
- Status: Granted
- Kind: B2
- Application number: 16/708,937
- Filed: 2019-12-10
- Published: 2021-06-10
- Granted: 2024-05-28
- Earliest claimed: 2019-12-10
- Assignees: Globus Medical, Inc.
- Applicants: Globus Medical, Inc.
- Inventors: Thomas Calloway; Weston Healy; Isaac Dulin; Dale Earle; Keiichi Matsuda
- Web page: https://patentgrep.com/patent/US11992373B2

## CPC

- A61B 90/50: Supports for surgical instruments, e.g. articulated arms
- A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 34/25: {User interfaces for surgical systems}
- A61B 34/37: Leader-follower robots (A61B34/35 takes precedence)
- A61B 90/36: {Image-producing devices or illumination devices not otherwise provided for}
- A61B 90/361: {Image-producing devices, e.g. surgical cameras}
- A61B 90/37: {Surgical systems with images on a monitor during operation}
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 27/017: {Head mounted}
- G02B 27/0172: {characterised by optical features}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06T 7/74: {involving reference images or patches}
- G06V 20/20: in augmented reality scenes
- A61B 2017/00199: {with a console, e.g. a control panel with a display}
- A61B 2017/00207: {with hand gesture control or hand gesture recognition}
- A61B 2017/00221: {with wireless transmission of data, e.g. by infrared radiation or radiowaves}
- A61B 2034/102: {Modelling of surgical devices, implants or prosthesis}
- A61B 2034/105: {Modelling of the patient, e.g. for ligaments or bones}
- A61B 2034/107: {Visualisation of planned trajectories or target regions}
- A61B 2034/2055: {Optical tracking systems}
- A61B 2034/2065: {Tracking using image or pattern recognition}
- A61B 2090/064: {for measuring force, pressure or mechanical tension}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/376: {using X-rays, e.g. fluoroscopy}
- A61B 2090/502: {Headgear, e.g. helmet, spectacles}
- G02B 2027/0178: {Eyeglass type (eyeglass details G02C)}
- G06T 2215/16: Using real world measurements to influence rendering
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A surgical system includes an AR headset and a AR headset controller. The AR headset is configured to be worn by a user during a surgical procedure and has a see-through display screen configured to display an AR image and to allow at least a portion of a real-world scene to pass therethrough for viewing by the user. The AR headset also includes an opacity filter positioned between at least one of the user's eyes and the real-world scene when the see-through display screen is viewed by the user. The opacity filter provides opaqueness to light from the real-world scene. The AR headset controller communicates with a navigation controller to receive navigation information from the navigation controller which provides guidance to the user during the surgical procedure on an anatomical structure, and generates the AR image based on the navigation information for display on the see-through display screen.

## Claims

### Claim 1

A surgical system comprising:
an augmented reality (AR) headset configured to be worn by a user during a surgical procedure and including:
an AR transmitter configured to project light for an AR image;
a see-through display screen configured to combine the projected AR image and a real-world scene for viewing by the user, and
an opacity filter positioned between at least one of the user's eyes and the real-world scene when the see-through display screen is viewed by the user, wherein the opacity filter is configured to provide opaqueness to light from the real-world scene; and
AR headset controller configured to communicate with a navigation controller to receive navigation information from the navigation controller which provides guidance to the user during the surgical procedure on an anatomical structure, and configured to generate the AR image based on the navigation information for display on the see-through display screen;
wherein the opacity filter is configured with at least two horizontally extending bands extending over both eyes, each band extending across the entire width of the see-through display screen, wherein an upper one of the at least two horizontally extending bands has a lower light transmissivity than a lower one of the at least two horizontally extending bands;
wherein the augmented reality headset is operationally coupled to a robotic system having a robotic arm that is controllable based on hand gesture commands that is sensed by the augmented reality headset.

### Claim 2 (depends on 1)

The surgical system of claim 1, wherein the see-through display screen includes a curved lens that extends laterally across and downward from a front part of the AR headset, and the opacity filter is on a surface of the curved lens.

### Claim 3 (depends on 1)

The surgical system of claim 1, wherein:
the opacity filter is configured as a laterally extending band;
the AR headset controller is configured to display in a region of the see-through display screen aligned with the laterally extending band of the opacity filter at least one of: 2D Axial, Sagital, and/or Coronal view images of patient anatomy; a planned and/or currently tracked surgical tool pose; graphical model of surgical implant location; video from a medical instrument; and user selectable menu items triggering operations controlling medical equipment, and to display in another region of the see-through display screen that is not aligned with the laterally extending band of the opacity filter at least one of: a 3D graphical model of the anatomical structure and surgical planning information; 3D graphical model of a surgical instrument; animated 3D graphical model of a surgical instrument displayed with a pose relative to a graphical model of the anatomical structure that is modified to track in real-time measured poses of the surgical instrument relative to the anatomical structure; and a graphical model of the anatomical structure and the navigation information from the navigation controller which provides visual guidance to the user during the surgical procedure on the anatomical structure.

### Claim 4 (depends on 1)

The surgical system of claim 1, wherein:
the opacity filter is configured as a laterally extending band,
wherein the see-through display screen includes a combiner configured to combine the light of the AR images projected from the AR light emitter and light from the real-world scene into a combined overlay image viewable by the user, and
wherein the laterally extending band of the opacity filter is on a surface of the combiner.

### Claim 5 (depends on 1)

The surgical system of claim 1, further comprising:
an AR emitter configured to project light of the AR images responsive to signalling from the AR headset controller,
wherein the see-through display screen includes a combiner configured to combine light of the AR images projected from the AR emitter and light from the real-world scene into a combined image viewable by the user, and
wherein the opacity filter includes an opacity gradient that extends on a surface of the combiner.

### Claim 6 (depends on 1)

The surgical system of claim 1, wherein for each of a plurality of types of AR images the AR headset controller is configured to select among a plurality of different regions of the see-through display screen for displaying the type of AR image based on a characteristic of the type of AR image and levels of opaqueness of regions of the opacity filter aligned with the different regions of the see-through display screen.

### Claim 7 (depends on 1)

The surgical system of claim 1, further comprising:
a camera configured to output video frames capturing the real-world scene,
wherein the AR headset controller is configured to determine brightness and/or color of a defined area within the video frames, and to control opacity of at least one area of the opacity filter based on the determined brightness and/or color of the defined area within the video frames.

### Claim 8 (depends on 1)

The surgical system of claim 1, further comprising:
a camera configured to output video frames capturing the real-world scene,
wherein the AR headset controller is configured to determine brightness and/or color of a defined area within the video frames, and to control brightness and/or color of the AR image displayed on the see-through display screen based on the determined brightness and/or color of the defined area within the video frames.

### Claim 9 (depends on 1)

The surgical system of claim 1, wherein the AR headset controller is configured to determine poses of the AR headset, and to control opacity of at least one area of the opacity filter based on the determined poses of the AR headset.

### Claim 10 (depends on 9)

The surgical system of claim 9, wherein the AR headset controller is configured to determine the poses of the AR headset relative to a surgical site containing the anatomical structure, and to control opacity of at least one area of the opacity filter based on the determined poses of the AR headset relative to the surgical site.

### Claim 11 (depends on 1)

The surgical system of claim 1, further comprising:
a camera configured to output video frames capturing the real-world scene,
wherein the AR headset controller is configured to identify an object within the video frames, determine brightness and/or color of the identified object within the video frames, determine poses of the AR headset relative to a real-world object corresponding to the identified object within the video frames, and to control opacity of at least one area of the opacity filter based on the determined brightness and/or color of the identified object within the video frames and based on the determined poses.

### Claim 12 (depends on 1)

The surgical system of claim 1, further comprising:
a camera configured to output video frames capturing the real-world scene,
wherein the AR headset controller is configured to identify an object within the video frames, determine brightness and/or color of the identified object within the video frames, determine poses of the AR headset relative to a real-world object corresponding to the identified object within the video frames, and to control brightness and/or color of the AR image displayed on the see-through display screen based on the determined brightness and/or color of the identified object within the video frames and based on the determined poses.

### Claim 13 (depends on 1)

The surgical system of claim 1, wherein:
the opacity filter includes a plurality of pixels arranged in a grid and electrically controllable to adjust pixel opacity to light transmissivity; and
wherein the AR headset controller is configured to electrically control the opacity filter to provide an increased opacity for at least some pixels that are selected among the grid based on being aligned with or offset a distance from at least a portion of a shape of an AR image displayed on the see-through display screen.

### Claim 14 (depends on 13)

The surgical system of claim 13, wherein the AR headset controller is configured to electrically control the opacity filter to provide an increased opacity for at least some pixels that are selected among the grid to form a shape that is defined based on a perimeter of the shape of AR image displayed on the see-through display screen.

### Claim 15 (depends on 13)

The surgical system of claim 13, wherein the AR headset controller is configured to electrically control the opacity filter to provide a region of pixels selected among the grid with a relatively high opacity based on being aligned with at least a portion of the shape of the AR image displayed on the see-through display screen, and to provide another region of pixels selected among the grid with a gradual transition in opacity based on being peripherally outside the shape of the AR image displayed on the see-through display screen.

### Claim 16

An augmented reality (AR) headset comprising:
an AR transmitter configured to project light for an AR image;
a see-through display screen configured to combine the projected AR image and a real-world scene for viewing by the user; and
an opacity filter is configured to be positioned between at least one of the user's eyes and the real-world scene while the user is wearing the AR headset to view the see-through display screen, wherein the opacity filter is configured to provide opaqueness to light from the real-world scene,
wherein the opacity filter is configured with at least two horizontally extending bands extending over both eyes, each band extending across the entire width of the see-through display screen, wherein an upper one of the at least two horizontally extending bands has a lower light transmissivity than a lower one of the at least two horizontally extending bands;
wherein the augmented reality headset is operationally coupled to a robotic system having a robotic arm, the robotic system being controllable based on hand gesture commands that is sensed by the augmented reality headset.

### Claim 17 (depends on 16)

The AR headset of claim 16, further comprising:
an AR headset controller configured to communicate with a navigation controller, wherein the AR headset controller is configured to receive navigation information from the navigation controller which provides visual guidance to the user during a surgical procedure, and to generate the AR image based on the navigation information for display on the see-through display screen,
wherein the see-through display screen includes a curved lens that extends laterally across and downward from a front bar of the AR headset, the opacity filter is configured as a laterally extending band on a surface of the curved lens, and the AR headset controller is configured to display in a region of the see-through display screen aligned with the laterally extending band of the opacity filter AR images that are generated using one set of types of navigation information and to display in another region of the see-through display screen that is not aligned with the laterally extending band of the opacity filter AR images that are generated using a different set of types of navigation information.

### Claim 18 (depends on 16)

The AR headset of claim 16, further comprising:
an AR headset controller configured to communicate with a navigation controller, wherein the AR headset controller is configured to receive navigation information from the navigation controller which provides visual guidance to the user during a surgical procedure, and to generate the AR image based on the navigation information for display on the see-through display screen; and
an AR emitter configured to project light of the AR images responsive to signalling from the AR headset controller,
wherein the see-through display screen includes a combiner configured to combine light of the AR images projected from the AR emitter and light from the real-world scene into a combined image viewable by the user, and
wherein the opacity filter is on a surface of the combiner.

### Claim 19 (depends on 16)

The AR headset of claim 16, further comprising:
a camera configured to output video frames capturing the real-world scene; and
an AR headset controller configured to
communicate with a navigation controller to receive navigation information providing guidance to the user during a surgical procedure,
generate the AR image based on the navigation information for display on the see-through display screen,
determine brightness and/or color of a defined area within the video frames, and control opacity of at least one area of the opacity filter based on the determined brightness and/or color of the defined area within the video frames.

### Claim 20 (depends on 16)

The AR headset of claim 16, further comprising:
A camera configured to output video frames capturing the real-world scene; and
an AR headset controller configured to
communicate with a navigation controller to receive navigation information providing guidance to the user during a surgical procedure,
generate the AR image based on the navigation information for display on the see-through display screen,
determine brightness and/or color of a defined area within the video frames, and control brightness and/or color of the AR image displayed on the see-through display screen based on the determined brightness and/or color of the defined area within the video frames.