# US 11,439,469 B2 · Virtual guidance for orthopedic surgical procedures

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,439,469 B2
- Document ID: US11439469B2
- Status: Granted
- Kind: B2
- Application number: 17/117,917
- Filed: 2020-12-10
- Published: 2021-04-01
- Granted: 2022-09-13
- Earliest claimed: 2018-06-19
- Assignees: Howmedica Osteonics Corp.
- Applicants: Howmedica Osteonics Corp.
- Inventors: Sergii Poltaretskyi; Jean Chaoui; Damien Cariou
- Web page: https://patentgrep.com/patent/US11439469B2

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G16H 20/40: relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
- A61B 5/1114: {Tracking parts of the body}
- A61B 5/1121: {Determining geometric values, e.g. centre of rotation or angular range of movement}
- A61B 5/1122: {of movement trajectories}
- A61B 5/1127: {using markers}
- A61B 5/681: {Wristwatch-type devices}
- A61B 17/142: {with reciprocating saw blades, e.g. with cutting edges at the distal end of the saw blades}
- A61B 17/15: Guides therefor
- A61B 17/1604: {Chisels; Rongeurs; Punches; Stamps}
- A61B 17/1626: {Control means; Display units}
- A61B 17/1659: {Surgical rasps, files, planes, or scrapers}
- A61B 17/1684: {for the shoulder}
- A61B 17/1703: {using imaging means, e.g. by X-rays}
- A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 34/25: {User interfaces for surgical systems}
- A61B 34/76: {Manipulators having means for providing feel, e.g. force or tactile feedback}
- A61B 90/08: {Accessories or related features not otherwise provided for}
- A61B 90/36: {Image-producing devices or illumination devices not otherwise provided for}
- A61B 90/361: {Image-producing devices, e.g. surgical cameras}
- A61B 90/37: {Surgical systems with images on a monitor during operation}
- A61B 90/39: {Markers, e.g. radio-opaque or breast lesions markers}
- A61B 90/90: Identification means for patients or instruments, e.g. tags
- A61B 90/92: coded with colour
- A61F 2/40: for shoulders
- A61F 2/4081: {Glenoid components, e.g. cups}
- G02B 27/0075: {with means for altering, e.g. increasing, the depth of field or depth of focus}
- G02B 27/017: {Head mounted}
- G02B 27/0172: {characterised by optical features}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 18/2163: {Partitioning the feature space}
- G06F 30/10: Geometric CAD
- G06F 30/20: Design optimisation, verification or simulation (optimisation, verification or simulation of circuit designs G06F30/30)
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/048: Activation functions
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/11: Region-based segmentation
- G06T 7/55: from multiple images
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G09B 5/06: with both visual and audible presentation of the material to be studied
- G09B 9/00: Simulators for teaching or training purposes (for the use of weapons F41; computing aspects G06; {protocols for games, networked simulations or virtual reality H04L67/131})
- G09B 19/003: {Repetitive work cycles; Sequence of movements}
- G09B 23/28: for medicine
- G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS
- G16H 30/40: for processing medical images, e.g. editing
- G16H 40/20: for the management or administration of healthcare resources or facilities, e.g. managing hospital staff or surgery rooms
- G16H 40/60: for the operation of medical equipment or devices
- G16H 40/63: for local operation
- G16H 40/67: for remote operation
- G16H 50/30: for calculating health indices; for individual health risk assessment
- G16H 50/50: for simulation or modelling of medical disorders
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- G16H 70/20: relating to practices or guidelines
- G16H 70/60: relating to pathologies
- G16H 80/00: ICT specially adapted for facilitating communication between medical practitioners or patients, e.g. for collaborative diagnosis, therapy or health monitoring
- H04N 13/122: Improving the three-dimensional [3D] impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues (H04N13/128 takes precedence)
- H04N 13/332: Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
- A61B 5/744: {Displaying an avatar, e.g. an animated cartoon character}
- A61B 17/151: {for corrective osteotomy}
- A61B 17/1775: {for the foot or ankle}
- A61B 17/1778: {for the shoulder}
- A61B 90/94: coded with symbols, e.g. text
- A61B 90/96: using barcodes
- A61B 90/98: using electromagnetic means, e.g. transponders
- A61B 2017/00017: {Electrical control of surgical instruments}
- A61B 2017/00115: {with audible or visual output}
- A61B 2017/00119: {alarm; indicating an abnormal situation}
- A61B 2017/00123: {and automatic shutdown}
- A61B 2034/102: {Modelling of surgical devices, implants or prosthesis}
- A61B 2034/104: {Modelling the effect of the tool, e.g. the effect of an implanted prosthesis or for predicting the effect of ablation or burring}
- A61B 2034/105: {Modelling of the patient, e.g. for ligaments or bones}
- A61B 2034/107: {Visualisation of planned trajectories or target regions}
- A61B 2034/108: {Computer aided selection or customisation of medical implants or cutting guides}
- A61B 2034/2048: {using an accelerometer or inertia sensor}
- A61B 2034/2051: {Electromagnetic tracking systems}
- A61B 2034/2055: {Optical tracking systems}
- A61B 2034/2057: {Details of tracking cameras}
- A61B 2034/2065: {Tracking using image or pattern recognition}
- A61B 2034/2068: {using pointers, e.g. pointers having reference marks for determining coordinates of body points}
- A61B 2034/252: {indicating steps of a surgical procedure}
- A61B 2034/254: {being adapted depending on the stage of the surgical procedure}
- A61B 2034/258: {providing specific settings for specific users}
- A61B 2090/062: {penetration depth}
- A61B 2090/067: {for measuring angles}
- A61B 2090/0801: {Prevention of accidental cutting or pricking}
- A61B 2090/0807: {Indication means}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/366: {using projection of images directly onto the body}
- A61B 2090/367: {creating a 3D dataset from 2D images using position information}
- A61B 2090/368: {changing the image on a display according to the operator's position}
- A61B 2090/372: {Details of monitor hardware}
- A61B 2090/373: {using light, e.g. by using optical scanners}
- A61B 2090/374: {NMR or MRI}
- A61B 2090/3762: {using computed tomography systems [CT]}
- A61B 2090/378: {using ultrasound}
- A61B 2090/3937: {Visible markers}
- A61B 2090/3945: {Active visible markers, e.g. light emitting diodes}
- A61B 2090/397: {electromagnetic other than visible, e.g. microwave}
- A61B 2090/502: {Headgear, e.g. helmet, spectacles}
- A61B 2505/05: Surgical care
- A61B 2562/0219: Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
- A61F 2/32: for the hip
- A61F 2/38: for elbows or knees
- A61F 2/4202: {for ankles}
- A61F 2/4606: {of wrists or ankles; of hands, e.g. fingers; of feet, e.g. toes}
- A61F 2/4612: {of shoulders}
- A61F 2002/4011: {including proximal or total replacement of the humerus}
- A61F 2002/4633: {for selection of endoprosthetic joints or for pre-operative planning}
- A61F 2002/4658: {for measuring dimensions, e.g. length}
- A61F 2002/4668: {for measuring angles}
- G02B 2027/0141: {characterised by the informative content of the display}
- G02B 2027/0174: {holographic}
- G06F 3/0483: Interaction with page-structured environments, e.g. book metaphor
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/042: Knowledge-based neural networks; Logical representations of neural networks
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/126: Evolutionary algorithms, e.g. genetic algorithms or genetic programming
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 20/20: Ensemble learning
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20036: Morphological image processing
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30008: Bone
- G06T 2207/30052: Implant; Prosthesis
- G06T 2207/30204: Marker
- G06T 2210/41: Medical
- G06T 2219/2004: Aligning objects, relative positioning of parts
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- A61F: FILTERS IMPLANTABLE INTO BLOOD VESSELS; PROSTHESES; DEVICES PROVIDING PATENCY TO, OR PREVENTING COLLAPSING OF, TUBULAR STRUCTURES OF THE BODY, e.g. STENTS; ORTHOPAEDIC, NURSING OR CONTRACEPTIVE DEVICES; FOMENTATION; TREATMENT OR PROTECTION OF EYES OR EARS; BANDAGES, DRESSINGS OR ABSORBENT PADS; FIRST-AID KITS (dental prosthetics A61C)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An example method includes displaying, via a visualization device and overlaid on a portion of an anatomy of a patient viewable via the visualization device, a virtual model of the portion of the anatomy obtained from a virtual surgical plan for an orthopedic joint repair surgical procedure to attach a prosthetic to the anatomy; and displaying, via the visualization device and overlaid on the portion of the anatomy, a virtual guide that guides at least one of preparation of the anatomy for attachment of the prosthetic or attachment of the prosthetic to the anatomy.

## Claims

### Claim 1

A method comprising:
displaying, via a visualization device and overlaid on a portion of an anatomy of a patient viewable via the visualization device, a virtual model of the portion of the anatomy obtained from a virtual surgical plan for an orthopedic joint repair surgical procedure to attach a prosthetic to the anatomy;
displaying, via the visualization device and overlaid on the portion of the anatomy, a virtual guide that guides at least one of preparation of the anatomy for attachment of the prosthetic or attachment of the prosthetic to the anatomy; and
while a mechanical hand tool is being used to prepare the anatomy for attachment of the prosthetic:
determining a current depth of the mechanical hand tool;
determining a relationship between the current depth of the mechanical hand tool and a target depth obtained from the virtual surgical plan; and
outputting a representation of the relationship, wherein outputting the representation comprises either:
outputting, via the visualization device, an audible representation of the relationship; or
outputting a haptic representation of the relationship.

### Claim 2 (depends on 1)

The method of claim 1, wherein the current depth comprises one or more of:
a current drilling depth;
a current reaming depth; and
a current cutting depth.

### Claim 3 (depends on 1)

The method of claim 1, wherein outputting the representation further comprises displaying, via the visualization device, a graphical representation of the relationship and displaying the graphical representation of the relationship comprises one or more of:
displaying a numerical value of a difference between the current depth and the target depth; and
displaying an image of a location of the portion of the anatomy and a location of the mechanical hand tool relative to the portion of the anatomy.

### Claim 4 (depends on 1)

The method of claim 1, wherein:
outputting the representation further comprises displaying, via the visualization device, a graphical representation of the relationship,
outputting the representation comprises displaying, via the visualization device, a graphical representation of the relationship,
displaying the graphical representation of the relationship comprises displaying an image of a location of the portion of the anatomy and a location of the mechanical hand tool relative to the portion of the anatomy, and
the image is from a side view.

### Claim 5 (depends on 1)

The method of claim 1, wherein displaying the virtual guide comprises projecting the virtual guide via a see-through lens through which a user is able to view the anatomy of the patient.

### Claim 6 (depends on 5)

The method of claim 5, wherein the portion of the anatomy of the patient is viewable via one or more see-through lenses of the visualization device, the method further comprising displaying the virtual model and displaying the virtual guide via the see-through lenses, and wherein displaying the virtual guide comprises displaying one or more of:
a virtual axis; or
a virtual cutting surface.

### Claim 7 (depends on 1)

The method of claim 1, wherein displaying the virtual guide comprises:
displaying, via the visualization device, one or more synthesized views of the mechanical hand tool being used to work on the anatomy.

### Claim 8 (depends on 7)

The method of claim 7, wherein displaying the synthesized views comprises one or more of:
displaying a synthesized side view as including a graphical representation of a relationship between the mechanical hand tool and a target vector; or
displaying a synthesized top view as including a graphical representation of a relationship between the mechanical hand tool and a target vector.

### Claim 9 (depends on 1)

The method of claim 1, wherein determining the current depth comprises determining, based on data generated by one or more sensors of the visualization device, the current depth.

### Claim 10 (depends on 1)

The method of claim 1, wherein outputting the haptic representation of the relationship comprises causing the mechanical hand tool to vibrate when the current depth of the mechanical hand tool reaches the target depth.

### Claim 11 (depends on 1)

The method of claim 1, wherein outputting the audible representation comprises outputting a tone that increases in volume as the current depth of the mechanical hand tool approaches the target depth.

### Claim 12 (depends on 1)

The method of claim 1, wherein using the mechanical hand tool to prepare the anatomy for attachment of the prosthetic comprises removing portions of the anatomy using the mechanical hand tool.

### Claim 13 (depends on 1)

The method of claim 1, wherein the mechanical hand comprises one of:
a drill;
a saw; or
a compactor.

### Claim 14

A mixed reality system comprising:
a memory that stores at least a portion of a virtual surgical plan; and
one or more processors configured to:
display, via a visualization device and overlaid on a portion of an anatomy of a patient viewable via the visualization device, a virtual model of the portion of the anatomy obtained from a virtual surgical plan for an orthopedic joint repair surgical procedure to attach a prosthetic to the anatomy;
display, via the visualization device and overlaid on the portion of the anatomy, a virtual guide that guides at least one of preparation of the anatomy for attachment of the prosthetic or attachment of the prosthetic to the anatomy; and
while a mechanical hand tool is being used to prepare the anatomy for attachment of the prosthetic:
determine a current depth of the mechanical hand tool;
determine a relationship between the current depth of the mechanical hand tool and a target depth obtained from the virtual surgical plan; and
output a representation of the relationship, wherein, to output the representation, the one or more processors are configured to either:
output, via the visualization device, an audible representation of the relationship; or
output a haptic representation of the relationship.

### Claim 15 (depends on 14)

The mixed reality system of claim 14, wherein the current depth comprises one or more of:
a current drilling depth;
a current reaming depth; and
a current cutting depth.

### Claim 16 (depends on 14)

The mixed reality system of claim 14, wherein, to output the representation, the one or more processors are further configured to display, via the visualization device, a graphical representation of the relationship, and wherein, to display the graphical representation of the relationship, the one or more processors are configured to one or more of:
display a numerical value of a difference between the current depth and the target depth; and
display an image of a location of the portion of the anatomy and a location of the mechanical hand tool relative to the portion of the anatomy.

### Claim 17 (depends on 14)

The mixed reality system of claim 14, wherein:
to output the representation, the one or more processors are configured to display, via the visualization device, a graphical representation of the relationship,
to output the representation, the one or more processors are configured to display, via the visualization device, a graphical representation of the relationship,
to display the graphical representation of the relationship, the one or more processors are configured to display an image of a location of the portion of the anatomy and a location of the mechanical hand tool relative to the portion of the anatomy, and
the image is from a side view.

### Claim 18 (depends on 14)

The mixed reality system of claim 14, wherein, to display the virtual guide, the one or more processors are configured to project the virtual guide via a see-through lens through which a user is able to view the anatomy of the patient.

### Claim 19 (depends on 18)

The mixed reality system of claim 18, wherein the portion of the anatomy of the patient is viewable via one or more see-through lenses of the visualization device, and wherein the one or more processors are further configured to displaying the virtual model and displaying the virtual guide via the see-through lenses.

### Claim 20 (depends on 14)

The mixed reality system of claim 14, wherein, to display the virtual guide, the one or more processors are configured to:
display, via the visualization device, one or more synthesized views of the mechanical hand tool being used to work on the anatomy.

### Claim 21 (depends on 20)

The mixed reality system of claim 20, wherein, to display the synthesized views, the one or more processors are configured to one or more of:
display a synthesized side view as including a graphical representation of a relationship between the mechanical hand tool and a target vector; or
display a synthesized top view as including a graphical representation of a relationship between the mechanical hand tool and a target vector.

### Claim 22 (depends on 14)

The mixed reality system of claim 14, wherein, to determine the current depth, the one or more processors are configured to determine, based on data generated by one or more sensors of the visualization device, the current depth.

### Claim 23

A computer-readable storage medium storing instructions that, when executed, cause one or more processors of a mixed reality system to:
display, via a visualization device and overlaid on a portion of an anatomy of a patient viewable via the visualization device, a virtual model of the portion of the anatomy obtained from a virtual surgical plan for an orthopedic joint repair surgical procedure to attach a prosthetic to the anatomy;
display, via the visualization device and overlaid on the portion of the anatomy, a virtual guide that guides at least one of preparation of the anatomy for attachment of the prosthetic or attachment of the prosthetic to the anatomy;
while a mechanical hand tool is being used to prepare the anatomy for attachment of the prosthetic:
determine a current depth of the mechanical hand tool;
determine a relationship between the current depth of the mechanical hand tool and a target depth obtained from the virtual surgical plan; and
output a representation of the relationship, wherein the instructions that cause the one or more processors to output the representation comprise instructions that cause the one or more processors to either:
output, via the visualization device, an audible representation of the relationship; or
output a haptic representation of the relationship.