# US 12,594,141 B2 · Systems, methods, and media for presenting biophysical simulations in an interactive mixed reality environment > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,594,141 B2 - Document ID: US12594141B2 - Status: Granted - Kind: B2 - Application number: 18/566,463 - Filed: 2022-06-03 - Published: 2024-08-08 - Granted: 2026-04-07 - Earliest claimed: 2021-06-03 - Assignees: Case Western Reserve University - Applicants: Case Western Reserve University - Inventors: Cameron McIntyre; Angela Noecker; Jeffrey Mlakar; Mark Griswold - Web page: https://patentgrep.com/patent/US12594141B2 ## CPC - A61B 90/37: {Surgical systems with images on a monitor during operation} - A61B 5/0042: {for the brain} - A61B 5/055: involving electronic [EMR] or nuclear [NMR] magnetic resonance, e.g. magnetic resonance imaging - A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations - A61B 90/361: {Image-producing devices, e.g. surgical cameras} - A61N 1/0534: {Electrodes for deep brain stimulation} - G02B 27/017: {Head mounted} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS - G16H 40/63: for local operation - G16H 50/50: for simulation or modelling of medical disorders - A61B 2017/00203: {with speech control or speech recognition} - A61B 2017/00207: {with hand gesture control or hand gesture recognition} - A61B 2017/00216: {with eye tracking or head position tracking control} - A61B 2034/102: {Modelling of surgical devices, implants or prosthesis} - A61B 2034/104: {Modelling the effect of the tool, e.g. the effect of an implanted prosthesis or for predicting the effect of ablation or burring} - A61B 2034/105: {Modelling of the patient, e.g. for ligaments or bones} - A61B 2034/107: {Visualisation of planned trajectories or target regions} - A61B 2034/2048: {using an accelerometer or inertia sensor} - A61B 2034/2074: {Interface software} - A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image} - A61B 2090/372: {Details of monitor hardware} - A61B 2090/374: {NMR or MRI} - A61B 2090/502: {Headgear, e.g. helmet, spectacles} - G16H 30/40: for processing medical images, e.g. editing - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - A61N: ELECTROTHERAPY; MAGNETOTHERAPY; RADIATION THERAPY; ULTRASOUND THERAPY (measurement of bioelectric currents A61B; surgical instruments, devices or methods for transferring non-mechanical forms of energy to or from the body A61B18/00; anaesthetic apparatus in general A61M; incandescent lamps H01K; infrared radiators for heating H05B) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA ## Abstract Systems, methods, and media for presenting biophysical simulations in an interactive mixed reality environment are provided. In some embodiments, a system comprises: a head mounted display comprising: a transparent display; sensors; and a processor programmed to: receive medical imaging data associated with a subject; receive, from a server, information useable to visualize a simulation of biophysical processes and a subject-specific anatomical model based on the medical imaging data; cause a visualization of the simulation to be presented, via the transparent display, in connection with the medical imaging data and an instrument in a first position; receive, from the server, updated information useable to visualize an updated simulation with the instrument in a second position; and cause a visualization of the updated simulation to be presented with the instrument presented in the second position. ## Claims ### Claim 1 A system for presenting biophysical simulations in an interactive mixed reality environment, comprising: a head mounted display comprising: a transparent display; a plurality of sensors; and at least one processor, wherein the at least one processor is programmed to: receive medical imaging data associated with a subject, the medical imaging data comprising T1-weighted magnetic resonance imaging (MRI) data, T2-weighted MRI data, or computed tomography (CT) data corresponding to the subject's brain tissue; receive, from a server, information useable to visualize a simulation of one or more biophysical processes associated with the subject in connection with a subject-specific anatomical model associated with the subject based on the medical imaging data with an instrument in a first position, wherein the server generates the subject-specific anatomical model based on the medical imaging data by using a nonlinear transformation matrix and a warp field with an anatomical atlas; cause a visualization of the simulation to be presented, via the transparent display, in connection with the medical imaging data with the instrument presented in the first position; receive, from the server, updated information useable to visualize an updated simulation of the one or more biophysical processes in connection with the subject-specific anatomical model based on the medical imaging data with the instrument in a second position; and cause a visualization of the updated simulation to be presented, via the transparent display, in connection with the medical imaging data with the instrument presented in the second position. ### Claim 2 (depends on 1) The system of claim 1, wherein the visualization includes a representation of the instrument. ### Claim 3 (depends on 1) The system of claim 1, wherein the instrument is a deep brain stimulation (DBS) electrode comprising multiple electrode contacts. ### Claim 4 (depends on 3) The system of claim 3, wherein the at least one processor is further programmed to: receive, via an input device, input to manipulate an activation state of one or more of the contacts of the DBS electrode. ### Claim 5 (depends on 1) The system of claim 1, wherein the instrument comprises; a stereo-encephalography (SEEG) electrode; a convection enhanced delivery (CED) probe; or a laser interstitial thermal therapy (LITT) probe. ### Claim 6 (depends on 1) The system of claim 1, wherein the at least one processor is further programmed to: receive, via an input device, input to manipulate a position of the instrument; and transmit, to the server, instructions based on the input. ### Claim 7 (depends on 6) The system of claim 6, wherein: the head mounted display further comprises the input device; the input device comprises a sensor; and the at least one processor is further programmed to: receive, from the sensor, information indicative of a position of an index finger of a wearer of the head mounted display, wherein the input to manipulate the position of the instrument comprises movement of the index finger from a first position to a second position. ### Claim 8 (depends on 1) The system of claim 1, wherein the at least one processor is further programmed to: transmit, to the server, instructions to adjust the instrument to the second position. ### Claim 9 (depends on 1) The system of claim 1, wherein the at least one processor is further programmed to: receive, via an input device, input to manipulate a position of a portion of the subject-specific anatomical model. ### Claim 10 (depends on 1) The system of claim 1, wherein the at least one processor is further programmed to: transmit, to the server, instructions to adjust a position of a portion of the subject-specific anatomical model. ### Claim 11 (depends on 1) The system of claim 1, wherein the at least one processor is further programmed to: receive, via an input device, input to change a portion of the simulation that is visualized. ### Claim 12 (depends on 1) The system of claim 1, wherein the at least one processor is further programmed to: transmit, to the server, instructions to change a portion of the simulation that is visualized. ### Claim 13 (depends on 1) The system of claim 1, wherein the anatomical atlas comprises a brain atlas. ### Claim 14 A method for presenting biophysical simulations in an interactive mixed reality environment, comprising: receiving medical imaging data associated with a subject, the medical imaging data comprising T1-weighted magnetic resonance imaging (MRI) data, T2-weighted MRI data, or computed tomography (CT) data corresponding to the subject's brain tissue; receiving, from a server, information useable to visualize a simulation of one or more biophysical processes associated with the subject in connection with a subject-specific anatomical model associated with the subject based on the medical imaging data with an instrument in a first position, wherein the server generates the subject-specific anatomical model based on the medical imaging data by using a nonlinear transformation matrix and a warp field with an anatomical atlas; causing a visualization of the simulation to be presented, via a transparent display of a head mounted display device, in connection with the medical imaging data with the instrument presented in the first position; receiving, from the server, updated information useable to visualize an updated simulation of the one or more biophysical processes in connection with the subject-specific anatomical model based on the medical imaging data with the instrument in a second position; and causing a visualization of the updated simulation to be presented, via the transparent display, in connection with the medical imaging data with the instrument presented in the second position. ### Claim 15 (depends on 14) The method of claim 14, wherein the instrument is a deep brain stimulation (DBS) electrode comprising multiple electrode contacts, a convection enhanced delivery (CED) probe, or a laser interstitial thermal therapy (LITT) probe. ### Claim 16 (depends on 14) The method of claim 14, wherein the anatomical atlas comprises a brain atlas. ### Claim 17 A non-transitory computer readable medium containing computer executable instructions that, when executed by processing circuitry, cause the processing circuitry to: receive medical imaging data associated with a subject, the medical imaging data comprising computed tomography (CT) data corresponding to the subject's brain tissue; receive, from a server, information useable to visualize a simulation of one or more biophysical processes associated with the subject in connection with a subject-specific anatomical model associated with the subject based on the medical imaging data with an instrument in a first position, wherein the server generates the subject-specific anatomical model based on the medical imaging data by using a nonlinear transformation matrix and a warp field with an anatomical atlas; cause a visualization of the simulation to be presented, via a transparent display of a head mounted display device, in connection with the medical imaging data with the instrument presented in the first position; receive, from the server, updated information useable to visualize an updated simulation of the one or more biophysical processes in connection with the subject-specific anatomical model based on the medical imaging data with the instrument in a second position; and cause a visualization of the updated simulation to be presented, via the transparent display, in connection with the medical imaging data with the instrument presented in the second position. ### Claim 18 (depends on 17) The non-transitory computer readable medium of claim 17, wherein the medical imaging data comprises T1-weighted magnetic resonance imaging (MRI) data or T2-weighted MRI data corresponding to the subject's brain tissue. ### Claim 19 (depends on 17) The non-transitory computer readable medium of claim 17, wherein the anatomical atlas comprises a brain atlas. ### Claim 20 (depends on 17) The non-transitory computer readable medium of claim 17, wherein the instrument is a deep brain stimulation (DBS) electrode comprising multiple electrode contacts, a convection enhanced delivery (CED) probe, or a laser interstitial thermal therapy (LITT) probe.