# US 12,112,843 B2 · Mixed reality-aided education related to orthopedic surgical procedures

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,843 B2
- Document ID: US12112843B2
- Status: Granted
- Kind: B2
- Application number: 17/117,654
- Filed: 2020-12-10
- Published: 2021-04-01
- Granted: 2024-10-08
- Earliest claimed: 2018-06-19
- Assignees: Howmedica Osteonics Corp.
- Applicants: Howmedica Osteonics Corp.
- Inventors: Sergii Poltaretskyi; Jean Chaoui; Damien Cariou
- Web page: https://patentgrep.com/patent/US12112843B2

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G16H 20/40: relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
- A61B 5/1114: {Tracking parts of the body}
- A61B 5/1121: {Determining geometric values, e.g. centre of rotation or angular range of movement}
- A61B 5/1122: {of movement trajectories}
- A61B 5/1127: {using markers}
- A61B 5/681: {Wristwatch-type devices}
- A61B 17/142: {with reciprocating saw blades, e.g. with cutting edges at the distal end of the saw blades}
- A61B 17/15: Guides therefor
- A61B 17/1604: {Chisels; Rongeurs; Punches; Stamps}
- A61B 17/1626: {Control means; Display units}
- A61B 17/1659: {Surgical rasps, files, planes, or scrapers}
- A61B 17/1684: {for the shoulder}
- A61B 17/1703: {using imaging means, e.g. by X-rays}
- A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 34/25: {User interfaces for surgical systems}
- A61B 34/76: {Manipulators having means for providing feel, e.g. force or tactile feedback}
- A61B 90/08: {Accessories or related features not otherwise provided for}
- A61B 90/36: {Image-producing devices or illumination devices not otherwise provided for}
- A61B 90/361: {Image-producing devices, e.g. surgical cameras}
- A61B 90/37: {Surgical systems with images on a monitor during operation}
- A61B 90/39: {Markers, e.g. radio-opaque or breast lesions markers}
- A61B 90/90: Identification means for patients or instruments, e.g. tags
- A61B 90/92: coded with colour
- A61F 2/40: for shoulders
- A61F 2/4081: {Glenoid components, e.g. cups}
- G02B 27/0075: {with means for altering, e.g. increasing, the depth of field or depth of focus}
- G02B 27/017: {Head mounted}
- G02B 27/0172: {characterised by optical features}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 18/2163: {Partitioning the feature space}
- G06F 30/10: Geometric CAD
- G06F 30/20: Design optimisation, verification or simulation (optimisation, verification or simulation of circuit designs G06F30/30)
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/048: Activation functions
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/11: Region-based segmentation
- G06T 7/55: from multiple images
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G09B 5/06: with both visual and audible presentation of the material to be studied
- G09B 9/00: Simulators for teaching or training purposes (for the use of weapons F41; computing aspects G06; {protocols for games, networked simulations or virtual reality H04L67/131})
- G09B 19/003: {Repetitive work cycles; Sequence of movements}
- G09B 23/28: for medicine
- G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS
- G16H 30/40: for processing medical images, e.g. editing
- G16H 40/20: for the management or administration of healthcare resources or facilities, e.g. managing hospital staff or surgery rooms
- G16H 40/60: for the operation of medical equipment or devices
- G16H 40/63: for local operation
- G16H 40/67: for remote operation
- G16H 50/30: for calculating health indices; for individual health risk assessment
- G16H 50/50: for simulation or modelling of medical disorders
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- G16H 70/20: relating to practices or guidelines
- G16H 70/60: relating to pathologies
- G16H 80/00: ICT specially adapted for facilitating communication between medical practitioners or patients, e.g. for collaborative diagnosis, therapy or health monitoring
- H04N 13/122: Improving the three-dimensional [3D] impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues (H04N13/128 takes precedence)
- H04N 13/332: Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
- A61B 5/744: {Displaying an avatar, e.g. an animated cartoon character}
- A61B 17/151: {for corrective osteotomy}
- A61B 17/1775: {for the foot or ankle}
- A61B 17/1778: {for the shoulder}
- A61B 90/94: coded with symbols, e.g. text
- A61B 90/96: using barcodes
- A61B 90/98: using electromagnetic means, e.g. transponders
- A61B 2017/00017: {Electrical control of surgical instruments}
- A61B 2017/00115: {with audible or visual output}
- A61B 2017/00119: {alarm; indicating an abnormal situation}
- A61B 2017/00123: {and automatic shutdown}
- A61B 2034/102: {Modelling of surgical devices, implants or prosthesis}
- A61B 2034/104: {Modelling the effect of the tool, e.g. the effect of an implanted prosthesis or for predicting the effect of ablation or burring}
- A61B 2034/105: {Modelling of the patient, e.g. for ligaments or bones}
- A61B 2034/107: {Visualisation of planned trajectories or target regions}
- A61B 2034/108: {Computer aided selection or customisation of medical implants or cutting guides}
- A61B 2034/2048: {using an accelerometer or inertia sensor}
- A61B 2034/2051: {Electromagnetic tracking systems}
- A61B 2034/2055: {Optical tracking systems}
- A61B 2034/2057: {Details of tracking cameras}
- A61B 2034/2065: {Tracking using image or pattern recognition}
- A61B 2034/2068: {using pointers, e.g. pointers having reference marks for determining coordinates of body points}
- A61B 2034/252: {indicating steps of a surgical procedure}
- A61B 2034/254: {being adapted depending on the stage of the surgical procedure}
- A61B 2034/258: {providing specific settings for specific users}
- A61B 2090/062: {penetration depth}
- A61B 2090/067: {for measuring angles}
- A61B 2090/0801: {Prevention of accidental cutting or pricking}
- A61B 2090/0807: {Indication means}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/366: {using projection of images directly onto the body}
- A61B 2090/367: {creating a 3D dataset from 2D images using position information}
- A61B 2090/368: {changing the image on a display according to the operator's position}
- A61B 2090/372: {Details of monitor hardware}
- A61B 2090/373: {using light, e.g. by using optical scanners}
- A61B 2090/374: {NMR or MRI}
- A61B 2090/3762: {using computed tomography systems [CT]}
- A61B 2090/378: {using ultrasound}
- A61B 2090/3937: {Visible markers}
- A61B 2090/3945: {Active visible markers, e.g. light emitting diodes}
- A61B 2090/397: {electromagnetic other than visible, e.g. microwave}
- A61B 2090/502: {Headgear, e.g. helmet, spectacles}
- A61B 2505/05: Surgical care
- A61B 2562/0219: Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
- A61F 2/32: for the hip
- A61F 2/38: for elbows or knees
- A61F 2/4202: {for ankles}
- A61F 2/4606: {of wrists or ankles; of hands, e.g. fingers; of feet, e.g. toes}
- A61F 2/4612: {of shoulders}
- A61F 2002/4011: {including proximal or total replacement of the humerus}
- A61F 2002/4633: {for selection of endoprosthetic joints or for pre-operative planning}
- A61F 2002/4658: {for measuring dimensions, e.g. length}
- A61F 2002/4668: {for measuring angles}
- G02B 2027/0141: {characterised by the informative content of the display}
- G02B 2027/0174: {holographic}
- G06F 3/0483: Interaction with page-structured environments, e.g. book metaphor
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/042: Knowledge-based neural networks; Logical representations of neural networks
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/126: Evolutionary algorithms, e.g. genetic algorithms or genetic programming
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 20/20: Ensemble learning
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20036: Morphological image processing
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30008: Bone
- G06T 2207/30052: Implant; Prosthesis
- G06T 2207/30204: Marker
- G06T 2210/41: Medical
- G06T 2219/2004: Aligning objects, relative positioning of parts
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- A61F: FILTERS IMPLANTABLE INTO BLOOD VESSELS; PROSTHESES; DEVICES PROVIDING PATENCY TO, OR PREVENTING COLLAPSING OF, TUBULAR STRUCTURES OF THE BODY, e.g. STENTS; ORTHOPAEDIC, NURSING OR CONTRACEPTIVE DEVICES; FOMENTATION; TREATMENT OR PROTECTION OF EYES OR EARS; BANDAGES, DRESSINGS OR ABSORBENT PADS; FIRST-AID KITS (dental prosthetics A61C)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An example system for demonstrating at least one aspect of an orthopedic surgical procedure includes a first device configured to display a presentation to a first user while the first user is wearing the device, wherein the presentation includes virtual element(s) that are controllable by the first user while the first user is wearing the first device, and wherein the virtual element(s) comprise a 3-dimensional (3D) virtual representation of one or more anatomical features associated with the orthopedic surgical procedure; and a second device configured to display the presentation to a second user, wherein the virtual element(s) further include virtual pre-operative plan element(s) shown relative to the 3D virtual representation, virtual surgical guidance feature(s) shown relative to the 3D virtual representation, or surgical result(s) virtually illustrated on the 3D virtual representation so as to demonstrate aspect(s) of the orthopedic surgical procedure.

## Claims

### Claim 1

An educational system for demonstrating at least one aspect of an orthopedic shoulder surgery, the system comprising:
a first device configured to display a mixed reality presentation to a first user while the first user is wearing the first device, wherein the first user is a teacher, wherein the mixed reality presentation includes one or more virtual elements that are controllable by the first user while the first user is wearing the first device, and wherein the one or more virtual elements comprise a 3-dimensional (3D) virtual representation of one or more anatomical features associated with the orthopedic shoulder surgery; and
a second device configured to display the mixed reality presentation to a second user to educate the second user about the orthopedic shoulder surgery, wherein the second user is a student and the one or more virtual elements that are controllable by the first user are also controllable by the second user while the second user is wearing the second device,
wherein the one or more virtual elements further include one or more virtual pre-operative plan elements shown relative to the 3D virtual representation, one or more virtual surgical guidance features shown relative to the 3D virtual representation, or one or more surgical results virtually illustrated on the 3D virtual representation so as to demonstrate the at least one aspect of the orthopedic shoulder surgery to the second user, wherein the one or more virtual elements comprise the 3D virtual representation positioned relative to a physical bone model of an anatomical element or a corresponding anatomical feature of a cadaver, and wherein the one or more virtual elements relate to the orthopedic shoulder surgery such that the student can practice the orthopedic shoulder surgery via the second device and the teacher can train the student on the orthopedic shoulder surgery via the first device.

### Claim 2 (depends on 1)

The educational system of claim 1,
wherein the first device comprises a first visualization device that includes a first holographic display that displays first virtual elements to the first user, and wherein the first virtual elements are superimposed on a first real-world view through the first visualization device, and
wherein the second device comprises a second visualization device that includes a second holographic display that displays second virtual elements to the second user, and wherein the second virtual elements are superimposed on a second real-world view through the second visualization device.

### Claim 3 (depends on 1)

The educational system of claim 1, wherein the 3D virtual representation of the one or more anatomical features comprises a 3D virtual model of a human shoulder.

### Claim 4 (depends on 1)

The educational system of claim 1, wherein the 3D virtual representation of the one or more anatomical features comprises a 3D virtual illustration of a humeral head or a 3D virtual illustration of a glenoid.

### Claim 5 (depends on 1)

The educational system of claim 1, wherein the one or more virtual elements illustrate a virtual cutting axis relative to the 3D virtual representation.

### Claim 6 (depends on 1)

The educational system of claim 1, wherein the one or more virtual elements illustrate a virtual reaming axis relative to the 3D virtual representation.

### Claim 7 (depends on 1)

The educational system of claim 1, wherein the one or more virtual elements illustrate a virtual drilling axis relative to the 3D virtual representation.

### Claim 8 (depends on 1)

The educational system of claim 1, wherein the one or more virtual elements illustrate placement of a jig or a guide relative to the 3D virtual representation.

### Claim 9 (depends on 8)

The educational system of claim 8, wherein the one or more virtual elements illustrate a virtual axis relative to the jig or the guide.

### Claim 10 (depends on 1)

The educational system of claim 1, wherein the one or more virtual elements comprise surgical guidance information.

### Claim 11 (depends on 1)

The educational system of claim 1, wherein the one or more virtual elements illustrate a registration process for registering the 3D virtual representation to a physical model or the corresponding anatomical feature of the cadaver.

### Claim 12 (depends on 1)

The educational system of claim 1, wherein the one or more virtual elements include trialing information associated with a prepared implantation location for an implant component.

### Claim 13 (depends on 1)

The educational system of claim 1, wherein the one or more virtual elements illustrate a registration process for registering a depth aid element.

### Claim 14 (depends on 1)

The educational system of claim 1, wherein the one or more virtual elements comprise virtual training information or virtual visual aids on a use of automated tools that include closed loop control.

### Claim 15 (depends on 1)

The educational system of claim 1, wherein the one or more virtual elements comprise virtual information about range of motion.

### Claim 16 (depends on 1)

The educational system of claim 1, wherein the one or more virtual elements include a virtual pre-operative animation.

### Claim 17 (depends on 1)

The educational system of claim 1, wherein the one or more virtual elements include one or more virtual implant components positioned relative to the 3D virtual representation.

### Claim 18 (depends on 17)

The educational system of claim 17, wherein the first device and the second device each comprises a mixed reality device.

### Claim 19 (depends on 1)

The educational system of claim 1, wherein the first device or the second device include a haptic device.

### Claim 20

A method for demonstrating at least one aspect of an orthopedic shoulder surgery by a first user to educate a second user, wherein the first user is a teacher and the second user is a student, the method comprising:
presenting one or more virtual elements on a first device, wherein the first device is controllable by the first user wearing the first device, wherein the one or more virtual elements comprise a 3D virtual representation of one or more anatomical features associated with the orthopedic shoulder surgery; and
presenting the one or more virtual elements on a second device, wherein the second device is associated with the second user and configured to display the one or more virtual elements that are controllable by the first user wearing the first device, wherein the one or more virtual elements that are controllable by the first user are also controllable by the second user while the second user is wearing the second device,
wherein the one or more virtual elements further include one or more virtual pre-operative plan elements shown relative to the 3D virtual representation, one or more virtual surgical guidance features shown relative to the 3D virtual representation, or one or more surgical results virtually illustrated on the 3D virtual representation so as to demonstrate the at least one aspect of the orthopedic shoulder surgery, and wherein the one or more virtual elements relate to the orthopedic shoulder surgery such that the student can practice the orthopedic shoulder surgery via the second device and the teacher can train the student on the orthopedic shoulder surgery via the first device.

### Claim 21 (depends on 20)

The method of claim 20, wherein the first device and the second device each comprises a mixed reality device.

### Claim 22

A visualization device configured to educate a user about an orthopedic shoulder surgery, wherein the user is a student, the visualization device comprising:
a processor configured to generate one or more virtual elements; and
a mixed reality display screen configured to present a mixed reality presentation to the user, wherein the mixed reality presentation includes the one or more virtual elements,
wherein the one or more virtual elements are controllable by a different user wearing a different visualization device and comprise a 3D virtual representation of one or more anatomical features associated with the orthopedic shoulder surgery, and wherein the different user is a teacher and the one or more virtual elements that are controllable by the user are also controllable by the different user while the different user is wearing the different visualization device, and wherein the one or more virtual elements further include one or more virtual pre-operative plan elements shown relative to the 3D virtual representation, one or more virtual surgical guidance features shown relative to the 3D virtual representation, or one or more surgical results virtually illustrated on the 3D virtual representation such that the mixed reality presentation is configured to educate the user about the orthopedic shoulder surgery.

### Claim 23 (depends on 22)

The visualization device of claim 22, wherein the mixed reality display screen comprise a see-through holographic lens configured to display the one or more virtual elements as one or more holograms.

### Claim 24 (depends on 22)

The visualization device of claim 22, wherein the 3D virtual representation of the one or more anatomical features comprises a 3D virtual model of a human shoulder.

### Claim 25 (depends on 22)

The visualization device of claim 22, wherein the 3D virtual representation of the one or more anatomical features comprises a 3D virtual illustration of a humeral head or a 3D virtual illustration of a glenoid.

### Claim 26 (depends on 22)

The visualization device of claim 22, wherein the one or more virtual elements illustrate a virtual cutting axis relative to the 3D virtual representation.

### Claim 27 (depends on 22)

The visualization device of claim 22, wherein the one or more virtual elements illustrate a virtual reaming axis relative to the 3D virtual representation.

### Claim 28 (depends on 22)

The visualization device of claim 22, wherein the one or more virtual elements illustrate a virtual drilling axis relative to the 3D virtual representation.

### Claim 29 (depends on 22)

The visualization device of claim 22, wherein the one or more virtual elements illustrate a virtual jig or guide relative to the 3D virtual representation.

### Claim 30 (depends on 29)

The visualization device of claim 29, wherein the one or more virtual elements illustrate a virtual axis relative to the virtual jig or guide.

### Claim 31 (depends on 22)

The visualization device of claim 22, wherein the mixed reality presentation is controllable by the user wearing the visualization device and by the different user wearing the different visualization device.

### Claim 32 (depends on 22)

The visualization device of claim 22, wherein the mixed reality presentation includes the 3D virtual representation positioned relative to a physical model of an anatomical element or a corresponding anatomical feature of a cadaver.

### Claim 33 (depends on 32)

The visualization device of claim 32, wherein the visualization device is configured demonstrate a registration process to register the 3D virtual representation with the physical model of the anatomical element or the corresponding anatomical feature of the cadaver.

### Claim 34 (depends on 22)

The visualization device of claim 22, wherein the one or more virtual elements include one or more virtual implant components positioned relative to the 3D virtual representation.

### Claim 35 (depends on 22)

The visualization device of claim 22, wherein the visualization device further comprises a haptic device.

### Claim 36

A system comprising:
a plurality of physical models of an anatomical element, wherein the plurality of physical models are substantially identical;
a first visualization device configured to display a first mixed reality presentation to a first user while the first user is wearing the first visualization device, wherein the first user is a teacher, wherein the first mixed reality presentation includes one or more first virtual elements that are controllable by the first user while the first user is wearing the first visualization device, and wherein the one or more first virtual elements comprise a first 3D virtual representation of one or more anatomical features associated with an orthopedic shoulder surgery, and wherein the first 3D virtual representation is positioned relative to a first physical model of the plurality of physical models; and
a second visualization device configured to display a second mixed reality presentation to a second user while the second user is wearing the second visualization device, wherein the second user is a student, wherein the second mixed reality presentation includes one or more second virtual elements that are controllable by the second user while the second user is wearing the second visualization device, and wherein the one or more second virtual elements comprise a second 3D virtual representation of the one or more anatomical features associated with the orthopedic shoulder surgery, and wherein the second 3D virtual representation is positioned relative to a second physical model of the plurality of physical models, and wherein the first and second virtual elements relate to the orthopedic shoulder surgery such that the student can practice the orthopedic shoulder surgery via the second visualization device and the teacher can train the student on the orthopedic shoulder surgery via the first visualization device.

### Claim 37 (depends on 36)

The system of claim 36, wherein the first visualization device is configured to display and control the first mixed reality presentation and to display the second mixed reality presentation, and wherein the second visualization device is configured to display and control the second mixed reality presentation and to display the first mixed reality presentation.