# US 12,010,285 B2 · Augmented reality guidance for spinal surgery with stereoscopic displays

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,010,285 B2
- Document ID: US12010285B2
- Status: Granted
- Kind: B2
- Application number: 18/352,778
- Filed: 2023-07-14
- Published: 2023-11-09
- Granted: 2024-06-11
- Earliest claimed: 2014-12-30
- Assignees: OnPoint Medical, Inc.
- Applicants: OnPoint Medical, Inc.
- Inventors: Carlos Quiles Casas
- Web page: https://patentgrep.com/patent/US12010285B2

## CPC

- H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
- A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- G02B 27/0172: {characterised by optical features}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04N 13/156: Mixing image signals
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 13/279: the virtual viewpoint locations being selected by the viewers or determined by tracking
- H04N 13/296: Synchronisation thereof; Control thereof
- H04N 13/366: using viewer tracking
- A61B 1/00: Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes (examination of body cavities or body tracts using ultrasonic, sonic or infrasonic waves A61B8/12; endoscopic instruments for taking cell samples or for biopsy A61B10/04); Illuminating arrangements therefor (for the eyes A61B3/00)
- A61B 2034/2057: {Details of tracking cameras}
- A61B 2034/2065: {Tracking using image or pattern recognition}
- A61B 2090/371: {with simultaneous use of two cameras}
- G02B 2027/0134: {of stereoscopic type}
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G06F 2203/04804: Transparency, e.g. transparent or translucent windows
- G06T 2210/41: Medical
- G16Z 99/00: Subject matter not provided for in other main groups of this subclass
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G16Z: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS, NOT OTHERWISE PROVIDED FOR

## Abstract

Embodiments disclose a real-time surgery method and apparatus for displaying a stereoscopic augmented view of a patient from a static or dynamic viewpoint of the surgeon, which employs real-time three-dimensional surface reconstruction for preoperative and intraoperative image registration. Stereoscopic cameras provide real-time images of the scene including the patient. A stereoscopic video display is used by the surgeon, who sees a graphical representation of the preoperative or intraoperative images blended with the video images in a stereoscopic manner through a see-through display.

## Claims

### Claim 1

A system comprising:
a stereoscopic optical see-through head mounted display;
at least one computing system; and
at least one camera, at least one 3D scanner, or a combination thereof,
wherein the at least one computing system is configured to track at least a portion of a physical instrument, at least a portion of a physical spinal implant, at least a portion of a physical spine of a patient or a combination thereof using the at least one camera, the at least one 3D scanner, or the combination thereof,
wherein the at least one computing system is configured to generate a three-dimensional (3D) surface representation of: at least a portion of a surgical guide, at least a portion of the tracked physical instrument, at least a portion of the tracked physical spinal implant, at least a portion of the tracked physical spine, at least a portion of an anatomical model, at least one image of the tracked physical spine, or a combination thereof,
wherein the at least one computing system is configured to generate a two-dimensional (2D) display of: the at least portion of the surgical guide, the at least portion of the tracked physical instrument, the at least portion of the tracked physical spinal implant, the at least portion of the tracked physical spine, the at least portion of the anatomical model, the at least one image of the tracked physical spine, or a combination thereof,
wherein the at least one computing system is configured to generate a stereoscopic view, displayed by the stereoscopic optical see-through head mounted display, the stereoscopic view comprising the 3D surface representation and the 2D display,
wherein the at least one computing system is configured to superimpose the stereoscopic view of the 3D surface representation on a target portion of the physical spine,
wherein the at least one computing system is configured to adjust in real time the stereoscopic view responsive to movement of the stereoscopic optical see-through head mounted display,
wherein the at least one computing system is configured to adjust in real time the stereoscopic view responsive to movement of the physical spine from a first position to a second position, and
wherein the first and second positions of the physical spine of the patient are different,
wherein the system is configured to guide placement of the physical spinal implant.

### Claim 2 (depends on 1)

The system of claim 1, wherein the at least one camera, the at least one 3D scanner, or the combination thereof is head mounted with the stereoscopic optical see-through head mounted display.

### Claim 3 (depends on 1)

The system of claim 1, wherein the at least one camera, the at least one 3D scanner, or the combination thereof is separate from the stereoscopic optical see-through head mounted display.

### Claim 4 (depends on 1)

The system of claim 1, wherein the stereoscopic optical see-through head mounted display comprises at least one display device with at least one data holding subsystem, logic subsystem or combination thereof in a shared enclosure.

### Claim 5 (depends on 1)

The system of claim 1, wherein the computing system is configured for communicatively coupling with one or more computing devices.

### Claim 6 (depends on 1)

The system of claim 1, wherein the computing system is configured for wireless coupling with one or more computing devices.

### Claim 7 (depends on 1)

The system of claim 1, wherein the system further comprises at least one inertial measurement unit, wherein the at least one inertial measurement unit is attached to or integrated into the at least one camera, the at least one 3D scanner, the stereoscopic optical see-through head mounted display, the at least one computing system, the physical instrument, the physical spinal implant, or a combination thereof.

### Claim 8 (depends on 1)

The system of claim 1, wherein the system further comprises one or more markers, wherein the one or more markers comprise a marker configured to be attached to the physical instrument, a marker configured to be attached to the physical spinal implant, a marker configured to be attached to a bony structure of the physical spine or a combination thereof, and wherein the at least one computing system is configured to track the one or more markers using the at least one camera, the at least one 3D scanner or the combination thereof.

### Claim 9 (depends on 1)

The system of claim 1, wherein the system further comprises at least one radiopaque marker.

### Claim 10 (depends on 9)

The system of claim 9, wherein the at least one radiopaque marker is included in the at least one image of the tracked physical spine, wherein the at least one image of the tracked physical spine comprises a preoperative, an intraoperative, or a preoperative and an intraoperative image of the tracked physical spine of the patient.

### Claim 11 (depends on 10)

The system of claim 10, wherein the at least one computing system is configured to register the at least one image with the physical spine using the at least one radiopaque marker.

### Claim 12 (depends on 8)

The system of claim 8, wherein the one or more markers comprise a color marker, a reflective marker, a passive marker, an active marker, or a combination thereof.

### Claim 13 (depends on 1)

The system of claim 1, wherein the 2D display comprises at least one 2D graphical representation of the at least portion of the surgical guide, the at least portion of the tracked physical instrument, the at least portion of the tracked physical spinal implant, the at least portion of the tracked physical spine, the at least portion of the anatomical model, the at least one image of the tracked physical spine, or the combination thereof.

### Claim 14 (depends on 1)

The system of claim 1, wherein the 3D surface representation comprises at least one 3D graphical representation of the at least portion of the surgical guide, the at least portion of the tracked physical instrument, the at least portion of the tracked physical spinal implant, the at least portion of the tracked physical spine, the at least portion of the anatomical model, the at least one image of the tracked physical spine, or the combination thereof.

### Claim 15 (depends on 1)

The system of claim 1, wherein the computing system is configured to generate one or more alerts when the at least portion of the tracked physical instrument or the at least portion of the tracked physical spinal implant is within a predetermined zone, a predetermined depth, or a combination thereof.

### Claim 16 (depends on 15)

The system of claim 15, wherein the one or more alerts comprise a visual alert displayed by the stereoscopic optical see-through head mounted display.

### Claim 17 (depends on 16)

The system of claim 16, wherein the one or more alerts comprise a color change, a contrast change or a combination thereof of the at least portion of the surgical guide, the at least portion of the tracked physical instrument, the at least portion of the tracked physical spinal implant, the at least portion of the tracked physical spine, the at least portion of the anatomical model, the at least one image of the tracked physical spine or the combination thereof, displayed by the stereoscopic optical see-through head mounted display.

### Claim 18 (depends on 1)

The system of claim 1, wherein the at least one computing system is configured to superimpose the stereoscopic view of the 2D display on the target portion of the physical spine.

### Claim 19 (depends on 1)

The system of claim 1, wherein the at least one image of the tracked physical spine comprises a 2D image of the physical spine, a 3D image of the physical spine, or a combination thereof.

### Claim 20 (depends on 1)

The system of claim 1, wherein the at least one image of the tracked physical spine comprises a 2D image, wherein the 2D image comprises an x-ray, an ultrasound scan, a CT scan, an MM scan of the target portion of the spine or a combination thereof, or wherein the at least one image of the tracked physical spine comprises a 3D image, wherein the 3D image comprises an ultrasound scan, a CT scan, an MM scan of the target portion of the spine or a combination thereof, or wherein the at least one image of the tracked physical spine comprises a 2D image and a 3D image, wherein the 2D image and the 3D image comprise an x-ray, an ultrasound scan, a CT scan, or an MM scan of the target portion of the spine or a combination thereof.

### Claim 21 (depends on 1)

The system of claim 1, wherein the at least one image of the tracked physical spine comprises a 2D image, a 3D image, or a 2D image and a 3D image, wherein the 2D image, 3D image, or 2D image and 3D image comprises a CT scan, an Mill scan, an ultrasound scan, an x-ray image or a combination thereof.

### Claim 22 (depends on 1)

The system of claim 1, wherein the at least one computing system comprises the computing system configured to track the at least portion of the physical instrument, the at least portion of the physical spinal implant, the at least portion of the physical spine of the patient or combination thereof, the computing system configured to generate the 3D surface representation, the computing system configured to generate the 2D display, the computing system configured to generate the stereoscopic view, the computing system configured to superimpose the stereoscopic view of the 3D surface representation on the target portion of the physical spine, the computing system configured to adjust in real time the stereoscopic view responsive to movement of the stereoscopic optical see-through head mounted display, the computing system configured to adjust in real time the stereoscopic view responsive to movement of the physical spine from the first position to the second position, and wherein at least two of the computing systems are the same.

### Claim 23 (depends on 1)

The system of claim 1, wherein the at least one computing system comprises the computing system configured to track the at least portion of the physical instrument, the at least portion of the physical spinal implant, the at least portion of the physical spine of the patient or combination thereof, the computing system configured to generate the 3D surface representation, the computing system configured to generate the 2D display, the computing system configured to generate the stereoscopic view, the computing system configured to superimpose the stereoscopic view of the 3D surface representation on the target portion of the physical spine, the computing system configured to adjust in real time the stereoscopic view responsive to movement of the stereoscopic optical see-through head mounted display, the computing system configured to adjust in real time the stereoscopic view responsive to movement of the physical spine from the first position to the second position, and wherein at least two of the computing systems are different.

### Claim 24 (depends on 1)

The system of claim 1, further comprising at least one passive optical marker, active optical marker, inertial measurement unit, magnetic tracking device, electromagnetic tracking device, ultrasonic tracking device, mechanical tracking device, tracking device using inertial measurement units, navigation system or a combination thereof configured for tracking the position, orientation or position and orientation of the physical instrument, the physical device, the physical spine, the stereoscopic optical see-through head mounted display or a combination thereof.

### Claim 25 (depends on 1)

The system of claim 1, wherein the at least one computing system is configured to track the physical instrument, the physical spinal implant, the physical spine, the stereoscopic optical see-through head mounted display or a combination thereof in a coordinate system.

### Claim 26 (depends on 25)

The system of claim 25, wherein the coordinate system is the coordinate system of the at least one camera, the at least one 3D scanner or the combination thereof.

### Claim 27 (depends on 25)

The system of claim 25, wherein the coordinate system is the coordinate system of a marker configured to be attached to a bony structure of the physical spine.

### Claim 28 (depends on 1)

The system of claim 1, wherein the at least one camera, the at least one 3D scanner or the combination thereof is configured to use visible light, or wherein the at least one camera, the at least one 3D scanner or the combination thereof is configured to use infrared light.

### Claim 29 (depends on 1)

The system of claim 1, wherein the spinal implant comprises at least one of a screw, a pin, or a plate.

### Claim 30 (depends on 7)

The system of claim 7, wherein the computing system is configured to adjust in real time the at least one stereoscopic view responsive to movement of the stereoscopic optical see-through head mounted display using data from the at least one inertial measurement unit attached to or integrated into the stereoscopic optical see-through head mounted display.