# US 2026/0076754 A1 · System and method for rendering operative guidance overlay

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0076754 A1
- Document ID: US20260076754A1
- Status: Published application
- Kind: A1
- Application number: 19/395,111
- Filed: 2025-11-20
- Published: 2026-03-19
- Earliest claimed: 2018-09-21
- Assignees: Immersivetouch, Inc.
- Applicants: Immersivetouch, Inc.,
- Inventors: Jia LUO; Chris ORRIS; Pat BANERJEE
- Web page: https://patentgrep.com/patent/US20260076754A1

## CPC

- A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations
- A61B 34/25: {User interfaces for surgical systems}
- A61B 90/361: {Image-producing devices, e.g. surgical cameras}
- G02B 27/017: {Head mounted}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06T 7/75: {involving models}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G16H 20/40: relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
- G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS
- G16H 30/40: for processing medical images, e.g. editing
- G16H 40/63: for local operation
- G16H 40/67: for remote operation
- G16H 50/50: for simulation or modelling of medical disorders
- A61B 90/96: using barcodes
- A61B 2017/00203: {with speech control or speech recognition}
- A61B 2017/00207: {with hand gesture control or hand gesture recognition}
- A61B 2017/00216: {with eye tracking or head position tracking control}
- A61B 2034/102: {Modelling of surgical devices, implants or prosthesis}
- A61B 2034/105: {Modelling of the patient, e.g. for ligaments or bones}
- A61B 2034/107: {Visualisation of planned trajectories or target regions}
- A61B 2034/2055: {Optical tracking systems}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/372: {Details of monitor hardware}
- A61B 2090/3983: {Reference marker arrangements for use with image guided surgery}
- A61B 2090/502: {Headgear, e.g. helmet, spectacles}
- A61F 2/30942: {for designing or making customized prostheses, e.g. using templates, CT or NMR scans, finite-element analysis or CAD-CAM techniques}
- A61F 2002/30985: {using three dimensional printing [3DP]}
- A61F 2002/4633: {for selection of endoprosthetic joints or for pre-operative planning}
- B33Y 50/00: Data acquisition or data processing for additive manufacturing
- B33Y 80/00: Products made by additive manufacturing
- G02B 2027/0134: {of stereoscopic type}
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G06T 2207/30004: Biomedical image processing
- G06T 2207/30204: Marker
- G06T 2210/41: Medical
- G06T 2219/2004: Aligning objects, relative positioning of parts
- G06T 2219/2021: Shape modification
- H04N 19/70: characterised by syntax aspects related to video coding, e.g. related to compression standards
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- A61F: FILTERS IMPLANTABLE INTO BLOOD VESSELS; PROSTHESES; DEVICES PROVIDING PATENCY TO, OR PREVENTING COLLAPSING OF, TUBULAR STRUCTURES OF THE BODY, e.g. STENTS; ORTHOPAEDIC, NURSING OR CONTRACEPTIVE DEVICES; FOMENTATION; TREATMENT OR PROTECTION OF EYES OR EARS; BANDAGES, DRESSINGS OR ABSORBENT PADS; FIRST-AID KITS (dental prosthetics A61C)
- B33Y: ADDITIVE MANUFACTURING, i.e. MANUFACTURING OF THREE-DIMENSIONAL [3D] OBJECTS BY ADDITIVE DEPOSITION, ADDITIVE AGGLOMERATION OR ADDITIVE LAYERING, e.g. BY 3D PRINTING, STEREOLITHOGRAPHY OR SELECTIVE LASER SINTERING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Embodiments for rendering an operative guidance overlay in association with a patient are disclosed. A method includes receiving image data indicating an anatomical structure of the patient and a structure of operative instruments; generating virtual models based on the image data; receiving sensor data associated with a user, the operative instruments, or the patient; dynamically generating guidance overlay data associated with the anatomical structure of the patient, or the operative instruments based on the virtual models and the sensor data; and rendering the operative guidance overlay in association with the patient based on the guidance overlay data using a wearable augmented reality interface device. The virtual models include a virtual anatomical model of the anatomical structure of the patient, or virtual instrument models associated with the operative instruments. The rendering comprises continuous alignment of the virtual anatomical model with the patient and the virtual instrument models with the operative instruments.

## Claims

### Claim 1

A system comprising:
a wearable augmented reality interface device configured to present an operative guidance overlay within a field of view of a user;
a set of sensors arranged in association with the wearable augmented reality interface device, wherein the set of sensors is configured to measure sensor data associated with at least one of: the user, one or more operative instruments, or a patient, and transmit the measured sensor data to one or more processors; and
a computing device comprising a memory configured to store computer-executable instructions and the one or more processors configured to execute the computer-executable instructions to:
receive image data associated with the patient, wherein the image data indicates an anatomical structure of the patient and a structure of each of the one or more operative instruments;
generate one or more virtual models based on the image data, wherein the one or more virtual models comprise at least one of: a virtual anatomical model of the anatomical structure of the patient, or one or more virtual instrument models associated with the one or more operative instruments;
receive the sensor data associated with at least one of: the user, the one or more operative instruments, or the patient;
dynamically generate guidance overlay data associated with at least one of: the anatomical structure of the patient, or the one or more operative instruments, based on the one or more virtual models, and the sensor data, wherein the guidance overlay data indicates the operative guidance overlay comprising one or more guidance markers associated with the patient, the one or more operative instruments, or both; and
render, using the wearable augmented reality interface device, the operative guidance overlay in association with the patient based on the guidance overlay data, wherein the rendering comprises continuous alignment of the virtual anatomical model with the patient and the continuous alignment of the one or more virtual instrument models with corresponding operative instrument of the one or more operative instruments.

### Claim 2 (depends on 1)

The system of claim 1, wherein the one or more processors are further configured to:
identify and localize one or more spatial reference markers associated with the patient, the one or more operative instruments, or both, wherein the spatial reference markers comprise at least one of: a passive visual marker, a light-emitting optical marker, an electromagnetic marker, or any combination thereof;
spatially align at least one of: the virtual anatomical model with the anatomical structure of the patient, or the one or more virtual instrument models with the corresponding one or more operative instruments, based on sensor data;
periodically receive updated sensor data from the set of sensors; and
continuously maintain the spatial alignment of at least one of: the virtual anatomical model with the anatomical structure of the patient, or the one or more virtual instrument models with the corresponding one or more operative instruments, as the patient or the operative instruments move relative to a user's viewpoint, based on the updated sensor data.

### Claim 3 (depends on 2)

The system of claim 2, wherein the sensor data comprises at least one of: environment data, user interaction data associated with the user, orientation data associated with the user, pose data associated with the patient or the one or more operative instruments, or alignment data associated with the one or more spatial reference markers.

### Claim 4 (depends on 3)

The system of claim 3, wherein the user interaction data associated with the user comprises at least one of: a gesture, a voice command, an eye movement, or a manual input provided via user-operated controls.

### Claim 5 (depends on 3)

The system of claim 3, wherein the one or more processors are further configured to:
apply an algorithm to at least one of: the orientation data, or the pose data;
determine, based on application of the algorithm, latency-adjusted orientation data associated with the user and latency-adjusted pose data associated with the at least one of: the patient, or the one or more operative instruments; and
generate the guidance overlay data associated with the anatomical structure of the patient, the one or more operative instruments, or both, based on the latency-adjusted orientation data and the latency-adjusted pose data.

### Claim 6 (depends on 5)

The system of claim 5, wherein the one or more processors are further configured to:
generate an initial pose estimate of the operative guidance overlay based on the sensor data;
predict a pose at each new time step using a previous estimate and a dynamic model;
obtain a new measurement from the sensor data at each new time step;
update the initial pose estimate to determine an updated pose estimate by optimally blending the predicted pose and the new measurement, accounting for their respective uncertainties; and
render the within the field of view of the user based on the updated pose estimate, such that the field of view appears spatially stable with respect to the anatomical structure of the patient, or the one or more operative instruments, or both.

### Claim 7 (depends on 5)

The system of claim 5, wherein the one or more processors are further configured to:
determine action data associated with an action, wherein the action is performed by the user on the anatomical structure of the patient, and wherein the action data is determined based on the sensor data; and
update the orientation data or the pose data of at least one of: the virtual anatomical model, or the one or more virtual instrument models to align the virtual anatomical model with the anatomical structure of the patient and the one or more virtual instrument models with the corresponding one or more operative instruments, based on the action data and the image data.

### Claim 8 (depends on 1)

The system of claim 1, wherein the virtual anatomical model of the anatomical structure of the patient comprises one or more sets of layers, each of the one or more sets of layers correspond to an anatomical layer associated with a region of the anatomical structure, and wherein the one or more processors are further configured to:
receive visual parameter data associated with each of the one or more sets of layers, wherein the visual parameter data is associated with at least one of: transparency, color, transfer function, gradient, ambient occlusion, or sharpness; and
generate the guidance overlay data associated with the anatomical structure of the patient based on the visual parameter data.

### Claim 9

A method comprising:
receiving image data associated with a patient, wherein the image data indicates an anatomical structure of the patient and a structure of each of one or more operative instruments;
generating one or more virtual models based on the image data, wherein the one or more virtual models comprise at least one of: a virtual anatomical model of the anatomical structure of the patient, or one or more virtual instrument models associated with the one or more operative instruments;
receiving, from a set of sensors, sensor data associated with at least one of: a user, the one or more operative instruments, or the patient;
dynamically generating guidance overlay data associated with at least one of: the anatomical structure of the patient, or the one or more operative instruments, based on the one or more virtual models and the sensor data, wherein the guidance overlay data indicates an operative guidance overlay comprising one or more guidance markers associated with the patient, the one or more operative instruments, or both; and
rendering, using a wearable augmented reality interface device, the operative guidance overlay in association with the patient based on the guidance overlay data, wherein the rendering comprises continuous alignment of the virtual anatomical model with the patient and the continuous alignment of the one or more virtual instrument models with a corresponding operative instrument of the one or more operative instruments.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
identifying and localizing one or more spatial reference markers associated with the patient, the one or more operative instruments, or both, wherein the spatial reference markers comprise at least one of: a passive visual marker, a light-emitting optical marker, an electromagnetic marker, or any combination thereof;
spatially aligning at least one of: the virtual anatomical model with the anatomical structure of the patient, or the one or more virtual instrument models with the corresponding one or more operative instruments, based on sensor data;
periodically receiving updated sensor data from the set of sensors; and
continuously maintaining the spatial alignment of at least one of: the virtual anatomical model with the anatomical structure of the patient, or the one or more virtual instrument models with the corresponding one or more operative instruments, as the patient or the operative instruments move relative to a user's viewpoint, based on the updated sensor data.

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
applying an algorithm to at least one of: orientation data, or pose data;
determining, based on application of the algorithm, latency-adjusted orientation data associated with the user and latency-adjusted pose data associated with at least one of: the patient, or the one or more operative instruments; and
generating the guidance overlay data associated with the anatomical structure of the patient, the one or more operative instruments, or both, based on the latency-adjusted orientation data and the latency-adjusted pose data.

### Claim 12 (depends on 11)

The method of claim 11, further comprising:
generating an initial pose estimate of the operative guidance overlay based on the sensor data;
predicting a pose at each new time step using a previous estimate and a dynamic model;
obtaining a new measurement from the sensor data at each new time step;
updating the initial pose estimate to determine an updated pose estimate by optimally blending the predicted pose and the new measurement, accounting for their respective uncertainties; and
rendering the operative guidance overlay within a field of view of the user based on the updated pose estimate, such that the operative guidance overlay appears spatially stable with respect to the anatomical structure of the patient, or the one or more operative instruments, or both.

### Claim 13 (depends on 11)

The method of claim 11, further comprising:
determining action data associated with an action, wherein the action is performed by the user on the anatomical structure of the patient, and wherein the action data is determined based on the sensor data; and
updating the orientation data or the pose data of at least one of: the virtual anatomical model, or the one or more virtual instrument models to align the virtual anatomical model with the anatomical structure of the patient and the one or more virtual instrument models with the corresponding one or more operative instruments, based on the action data and the image data.

### Claim 14 (depends on 9)

The method of claim 9, wherein the virtual anatomical model of the anatomical structure of the patient comprises one or more sets of layers, each of the one or more sets of layers correspond to an anatomical layer associated with a region of the anatomical structure, and wherein the method further comprises:
receiving visual parameter data associated with each of the one or more sets of layers, wherein the visual parameter data is associated with at least one of: transparency, color, transfer function, gradient, ambient occlusion, metallic, or sharpness; and
generating the guidance overlay data associated with the anatomical structure of the patient based on the visual parameter data.

### Claim 15

A computer programmable product comprising a non-transitory computer-readable medium having stored thereon computer-executable instructions, which when executed by one or more processors, cause the one or more processors to carry out operations comprising:
receiving image data associated with a patient, wherein the image data indicates an anatomical structure of the patient and a structure of each of one or more operative instruments;
generating one or more virtual models based on the image data, wherein the one or more virtual models comprise at least one of: a virtual anatomical model of the anatomical structure of the patient, or one or more virtual instrument models associated with the one or more operative instruments;
receive, from a set of sensors, sensor data associated with at least one of: a user, the one or more operative instruments, or the patient;
dynamically generating guidance overlay data associated with at least one of: the anatomical structure of the patient, or the one or more operative instruments, based on the one or more virtual models and the sensor data, wherein the guidance overlay data indicates an operative guidance overlay comprising one or more guidance markers associated with the patient, the one or more operative instruments, or both; and
rendering, using a wearable augmented reality interface device, the operative guidance overlay in association with the patient based on the guidance overlay data, wherein the rendering comprises continuous alignment of the virtual anatomical model with the patient and the continuous alignment of the one or more virtual instrument models with a corresponding operative instrument of the one or more operative instruments.

### Claim 16 (depends on 15)

The computer programmable product of claim 15, wherein the operations further comprise:
identifying and localizing one or more spatial reference markers associated with the patient, the one or more operative instruments, or both, wherein the spatial reference markers comprise at least one of: a passive visual marker, a light-emitting optical marker, an electromagnetic marker, or any combination thereof;
spatially aligning at least one of: the virtual anatomical model with the anatomical structure of the patient, or the one or more virtual instrument models with the corresponding one or more operative instruments, based on sensor;
periodically receiving updated sensor data from the set of sensors; and
continuously maintaining the spatial alignment of at least one of: the virtual anatomical model with the anatomical structure of the patient, or the one or more virtual instrument models with the corresponding one or more operative instruments, as the patient or the operative instruments move relative to a user's viewpoint, based on the updated sensor data.

### Claim 17 (depends on 16)

The computer programmable product of claim 16, wherein the operations further comprise:
applying an algorithm to at least one of: orientation data, or pose data;
determining, based on application of the algorithm, latency-adjusted orientation data associated with the user and latency-adjusted pose data associated with the at least one of: the patient, or the one or more operative instruments; and
generating the guidance overlay data associated with the anatomical structure of the patient, the one or more operative instruments, or both, based on the latency-adjusted orientation data and the latency-adjusted pose data.

### Claim 18 (depends on 17)

The computer programmable product of claim 17, wherein the operations further comprise:
generating an initial pose estimate of the operative guidance overlay based on the sensor data;
predicting a pose at each new time step using a previous estimate and a dynamic model;
obtaining a new measurement from the sensor data at each new time step;
updating the initial pose estimate to determine an updated pose estimate by optimally blending the predicted pose and the new measurement, accounting for their respective uncertainties; and
rendering the operative guidance overlay within a field of view of the user based on the updated pose estimate, such that the operative guidance overlay appears spatially stable with respect to the anatomical structure of the patient, or the one or more operative instruments, or both.

### Claim 19 (depends on 16)

The computer programmable product of claim 16, wherein the operations further comprise:
determining action data associated with an action, wherein the action is performed by the user on the anatomical structure of the patient, and wherein the action data is determined based on the sensor data; and
updating orientation data or pose data of at least one of: the virtual anatomical model, or the one or more virtual instrument models to align the virtual anatomical model with the anatomical structure of the patient and the one or more virtual instrument models with the corresponding one or more operative instruments, based on the action data and the image data.

### Claim 20 (depends on 15)

The computer programmable product of claim 15, wherein the virtual anatomical model of the anatomical structure of the patient comprises one or more sets of layers, each of the one or more sets of layers correspond to an anatomical layer associated with a region of the anatomical structure, and wherein the operations further comprise:
receiving visual parameter data associated with each of the one or more sets of layers, wherein the visual parameter data is associated with at least one of: transparency, color, transfer function, gradient, ambient occlusion, metallic, or sharpness; and
generating the guidance overlay data associated with the anatomical structure of the patient based on the visual parameter data.