# US 12,225,181 B2 · Extended reality headset camera system for computer assisted navigation in surgery

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,225,181 B2
- Document ID: US12225181B2
- Status: Granted
- Kind: B2
- Application number: 18/483,105
- Filed: 2023-10-09
- Published: 2024-04-18
- Granted: 2025-02-11
- Earliest claimed: 2020-05-08
- Assignees: Globus Medical, Inc.
- Applicants: Globus Medical, Inc.
- Inventors: Weston Healy; Thomas Calloway; Norbert Johnson; Keerthighaan Kanagasegar
- Web page: https://patentgrep.com/patent/US12225181B2

## CPC

- A61B 90/36: {Image-producing devices or illumination devices not otherwise provided for}
- H04N 13/344: with head-mounted left-right displays
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 90/53: connected to the surgeon's body, e.g. by a belt
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 23/11: for generating image signals from visible and infrared light wavelengths
- H04N 23/12: with one sensor only
- H04N 23/21: from near infrared [NIR] radiation
- A61B 34/25: {User interfaces for surgical systems}
- A61B 34/30: Surgical robots
- A61B 2017/00207: {with hand gesture control or hand gesture recognition}
- A61B 2017/00734: {battery operated}
- A61B 2017/00991: {Telescopic means}
- A61B 2034/102: {Modelling of surgical devices, implants or prosthesis}
- A61B 2034/105: {Modelling of the patient, e.g. for ligaments or bones}
- A61B 2034/2048: {using an accelerometer or inertia sensor}
- A61B 2034/2057: {Details of tracking cameras}
- A61B 2034/2059: {Mechanical position encoders}
- A61B 2034/2072: {Reference field transducer attached to an instrument or patient}
- A61B 2090/064: {for measuring force, pressure or mechanical tension}
- A61B 2090/3616: {Magnifying glass}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/371: {with simultaneous use of two cameras}
- A61B 2090/372: {Details of monitor hardware}
- A61B 2090/376: {using X-rays, e.g. fluoroscopy}
- A61B 2090/3762: {using computed tomography systems [CT]}
- A61B 2090/3764: {with a rotating C-arm having a cone beam emitting source}
- A61B 2090/3945: {Active visible markers, e.g. light emitting diodes}
- A61B 2090/502: {Headgear, e.g. helmet, spectacles}
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A head-mounted extended reality (XR) display device includes a rigid mounting element coupled to a frame. The XR display device further includes right-side and left-side visible light cameras coupled to the rigid mounting element, right-side and left-side near-infrared (NIR) cameras coupled to the rigid mounting element, and an NIR light-emitting diode (LED) configured to illuminate a region within a field of view of the NIR cameras. The visible light cameras are configured to capture stereoscopic visible light images within a field of view of the user when the user is wearing the frame, and the NIR cameras are configured to capture stereoscopic NIR images within the field of view of the user when the user is wearing the frame.

## Claims

### Claim 1

A head-mounted extended reality (XR) display device comprising:
a frame configured to be worn by a user's head;
a rigid mounting element coupled to the frame;
a camera coupled to the rigid mounting element;
wherein the camera is configured to capture stereoscopic visible light images within a field of view of the user when the user is wearing the frame, and
a tracking system adapted to determine a pose of dynamic reference arrays attached to a patient and to surgical instruments based on stereoscopic visible light images received through the camera, the tracking system further adapted to determine hand gesture based commands based on stereoscopic NIR images received through the camera.

### Claim 2 (depends on 1)

The XR display device of claim 1, wherein the camera defines a field of view having a first center in a first direction extending away from the frame,
wherein the camera defines a field of view having a second center in a second direction extending away from the frame, and
wherein the first direction and the second direction intersect at an intersection point within a field of view of the user when the user is wearing the frame.

### Claim 3 (depends on 2)

The XR display device of claim 2, wherein the first direction and the second direction define an angle at the intersection point that is greater than 10 degrees and smaller than 15 degrees.

### Claim 4 (depends on 3)

The XR display device of claim 3, wherein the angle is substantially 12 degrees.

### Claim 5 (depends on 2)

The XR display device of claim 2, wherein the frame defines a substantially horizontal reference plane when the frame is being worn by the user, and
wherein the first direction and the second direction are angled downwardly with respect to the horizontal reference plane at a downward angle that is greater than 50 degrees and less than 60 degrees when the frame is being worn by the user.

### Claim 6 (depends on 5)

The XR display device of claim 5, wherein the downward angle is substantially 55 degrees.

### Claim 7 (depends on 1)

The XR display device of claim 1, further comprising:
an optical combiner attached to the frame and adapted to pass real world images toward the user's eyes;
a display mounted to the frame so as to project virtual graphical objects on to the optical combiner to be reflected back towards the user's eyes.

### Claim 8 (depends on 1)

The XR display device of claim 1, wherein the camera is configured to capture the stereoscopic visible light images at a first resolution, and to capture the stereoscopic NIR images at a second resolution less than the first resolution.

### Claim 9 (depends on 8)

The XR display device of claim 8, further comprising a loupe camera coupled to the frame, the loupe camera configured to capture color images within the field of view of the user at a third resolution greater than the first resolution.

### Claim 10

An extended reality (XR) camera assembly for a head-mounted display device comprising:
a rigid mounting element configured to be coupled to a head-wearable frame;
a visible light camera coupled to the rigid mounting element;
a near-infrared (NIR) camera coupled to the rigid mounting element;
at least one NIR light-emitting diode (LED) configured to illuminate a region within a field of view of the right-side NIR camera and the left-side NIR camera,
wherein the visible light camera is configured to capture stereoscopic visible light images within a field of view of a user when the user is wearing the frame, and
wherein the NIR camera is configured to capture stereoscopic NIR images within the field of view of the user when the user is wearing the frame,
a tracking system adapted to determine a pose of dynamic reference arrays attached to a patient and to surgical instruments based on stereoscopic visible light images received through the visible light camera, the tracking system further adapted to determine hand gesture based commands based on the stereoscopic NIR images received through the NIR cameras.

### Claim 11 (depends on 10)

The XR camera assembly of claim 10, wherein the visible light camera defines a field of view having a first center in a first direction extending away from the frame,
wherein the visible light camera defines a field of view having a second center in a second direction extending away from the frame, and
wherein the first direction and the second direction intersect at an intersection point within a field of view of the user when the user is wearing the frame.

### Claim 12 (depends on 11)

The XR camera assembly of claim 11, wherein the first direction and the second direction define an angle at the intersection point that is greater than 10 degrees and smaller than 15 degrees.

### Claim 13 (depends on 12)

The XR camera assembly of claim 12, wherein the angle is substantially 12 degrees.

### Claim 14 (depends on 11)

The XR camera assembly of claim 11, wherein the frame defines a substantially horizontal reference plane when the frame is being worn by the user, and
wherein the first direction and the second direction are angled downwardly with respect to the horizontal reference plane at a downward angle that is greater than 50 degrees and less than 60 degrees when the frame is being worn by the user.

### Claim 15 (depends on 14)

The XR camera assembly of claim 14, wherein the downward angle is substantially 55 degrees.

### Claim 16 (depends on 10)

The XR display device of claim 10, further comprising:
an optical combiner attached to the frame and adapted to pass real world images toward the user's eyes;
a display mounted to the frame so as to project virtual graphical objects on to the optical combiner to be reflected back towards the user's eyes.

### Claim 17 (depends on 10)

The XR camera assembly of claim 10, wherein the visible light camera and the is configured to capture the stereoscopic visible light images at a first resolution, and
wherein the NIR camera is configured to capture the stereoscopic NIR images at a second resolution less than the first resolution.

### Claim 18 (depends on 17)

The XR camera assembly of claim 17, further comprising a loupe camera coupled to the frame, the loupe camera configured to capture color images within the field of view of the user at a third resolution greater than the first resolution.

### Claim 19

A method of generating extended reality (XR) imagery comprising:
capturing a first plurality of visible light stereoscopic images of a scene within a field of view of a visible light cameras coupled to a rigid mounting element during a time period;
capturing a second plurality of near-infrared (NIR) stereoscopic images of a scene within a NIR cameras coupled to the rigid mounting element during the time period;
determining a pose of dynamic reference arrays attached to a patient and to a surgical instrument based on the first plurality of visible light stereoscopic images within a three-dimensional space; and
determining a hand gesture based commands based on the second plurality of NIR stereoscopic images within the three-dimensional space.

### Claim 20 (depends on 19)

The method of claim 19, wherein the pair of visible light cameras are disposed at an angle with respect to each other that is greater than 10 degrees and smaller than 15 degrees.