# US 12,002,171 B2 · Surgeon head-mounted display apparatuses > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,002,171 B2 - Document ID: US12002171B2 - Status: Granted - Kind: B2 - Application number: 17/935,585 - Filed: 2022-09-27 - Published: 2023-03-16 - Granted: 2024-06-04 - Earliest claimed: 2015-02-03 - Assignees: Globus Medical, Inc. - Applicants: Globus Medical, Inc. - Inventors: Kenneth Milton Jones; John Popoolapade; Thomas Calloway; Thierry Lemoine; Christian Jutteau; Christophe Bruzy; Yannick James; Joachim Laguarda; Dong-Mei Pei Xing; Sebastien Gorges; Paul Michael Yarin - Web page: https://patentgrep.com/patent/US12002171B2 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations - A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis - A61B 90/37: {Surgical systems with images on a monitor during operation} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/012: {Head tracking input arrangements} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - A61B 2034/105: {Modelling of the patient, e.g. for ligaments or bones} - A61B 2034/2055: {Optical tracking systems} - A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image} - A61B 2090/368: {changing the image on a display according to the operator's position} - A61B 2090/372: {Details of monitor hardware} - G06T 2207/30204: Marker - G06T 2210/41: Medical - G06T 2219/2016: Rotation, translation, scaling - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract An augmented reality surgical system includes a head mounted display (HMD) with a see-through display screen, a motion sensor, a camera, and computer equipment. The motion sensor outputs a head motion signal indicating measured movement of the HMD. The computer equipment computes the relative location and orientation of reference markers connected to the HMD and to the patient based on processing a video signal from the camera. The computer equipment generates a three dimensional anatomical model using patient data created by medical imaging equipment, and rotates and scales at least a portion of the three dimensional anatomical model based on the relative location and orientation of the reference markers, and further rotate at least a portion of the three dimensional anatomical model based on the head motion signal to track measured movement of the HMD. The rotated and scaled three dimensional anatomical model is displayed on the display screen. ## Claims ### Claim 1 An augmented reality head mounted display (HMD) system comprising: a housing; an adjustable headband attached to the HMD housing and configured to adjustably grasp a user's head; a partially transparent display screen extending downwardly from the housing; a display module adapted to generate video images to be projected onto the display screen to be combined with real world images for viewing by the user; a plurality of reference markers attached to the housing and positioned to be visible to a position tracking system; a computer equipment configured to: determine the location and orientation of the reference markers based on input from the position tracking system; generate a three dimensional anatomical image of a patient; rotate and scale the generated three dimensional anatomical image based on the determined location and orientation of the reference markers; and display through the display module the rotated and scaled image on the display screen for viewing by the user. ### Claim 2 (depends on 1) The system of claim 1, wherein the adjustable headband comprises: an adjustable side band extending from the housing and laterally around the user's head; a central band extending vertically from the housing and over the user's head. ### Claim 3 (depends on 2) The system of claim 2, wherein the central band is adapted to be length adjustable. ### Claim 4 (depends on 3) The system of claim 3, wherein one end of the central band is attached to the housing and the other end is attached to the adjustable side band. ### Claim 5 (depends on 4) The system of claim 4, wherein the adjustable side band includes a first lateral band extending from one side of the housing and a second lateral band extending from the other side of the housing. ### Claim 6 (depends on 1) The system of claim 1, wherein the adjustable headband comprises: an adjustable side band extending from the housing and laterally around the user's head; a central band extending vertically from the housing and over the user's head; a side band knob for adjustably fixing the side band length; and a central band knob for adjustably fixing the central band length. ### Claim 7 (depends on 1) The system of claim 1, further comprising at least one front facing camera adapted to be connected to a gesture interpretation module that interprets hand gestures of the user. ### Claim 8 (depends on 1) The system of claim 1, further comprising a plurality of patient tracking markers adapted to be attached to the patient and positioned to be visible to a position tracking system, wherein the computer equipment is configured to rotate and scale the generated three dimensional anatomical image based on the determined locations and orientations of the reference marker and the patient tracking markers. ### Claim 9 An augmented reality head mounted display (HMD) system comprising: a housing; an adjustable headband attached to the HMD housing and configured to adjustably grasp a user's head; a partially transparent display screen extending downwardly from the housing; a display module adapted to generate video images to be projected onto the display screen to be combined with real world images for viewing by the user; a plurality of patient tracking markers adapted to be attached to the patient and positioned to be visible to a position tracking system; a computer equipment configured to: determine the location and orientation of the patient tracking markers based on input from the position tracking system; generate a three dimensional anatomical image of a patient; rotate and scale the generated three dimensional anatomical image based on the determined location and orientation of the patient tracking markers; and display through the display module the rotated and scaled image on the display screen for viewing by the user. ### Claim 10 An augmented reality head mounted display (HMD) system comprising: an HMD including: a housing; an adjustable headband attached to the HMD housing and configured to adjustably grasp a user's head; a plurality of cameras configured to output video images; a plurality of reference markers attached to the housing and positioned to be visible to a camera tracking system; a partially transparent display screen extending downwardly from the housing; a display module adapted to generate video images to be projected onto the display screen to be combined with real world images for viewing by the user; a computer equipment configured to: determine the location and orientation of the reference markers based on input from the camera tracking system; generate a three dimensional anatomical image of a patient; rotate and scale the generated three dimensional anatomical image based on the determined location and orientation of the reference markers; and display through the display module the rotated and scaled image on the display screen for viewing by the user. ### Claim 11 (depends on 10) The system of claim 10, wherein the adjustable headband comprises: an adjustable side band extending from the housing and laterally around the user's head; a central band extending vertically from the housing and over the user's head. ### Claim 12 (depends on 11) The system of claim 11, wherein the central band is adapted to be length adjustable. ### Claim 13 (depends on 12) The system of claim 12, wherein one end of the central band is attached to the housing and the other end is attached to the adjustable side band. ### Claim 14 (depends on 13) The system of claim 13, wherein the adjustable side band includes a first lateral band extending from one side of the housing and a second lateral band extending from the other side of the housing. ### Claim 15 (depends on 10) The system of claim 10, wherein the adjustable headband comprises: an adjustable side band extending from the housing and laterally around the user's head; a central band extending vertically from the housing and over the user's head; a side band knob for adjustably fixing the side band length; and a central band knob for adjustably fixing the central band length. ### Claim 16 (depends on 10) The system of claim 10, further comprising at least one front facing camera adapted to be connected to a gesture interpretation module that interprets hand gestures of the user. ### Claim 17 (depends on 10) The system of claim 10, further comprising a plurality of patient tracking markers adapted to be attached to the patient and positioned to be visible to the camera tracking system, wherein the computer equipment is configured to rotate and scale the generated three dimensional anatomical image based on the determined locations and orientations of the reference marker and the patient tracking markers.