# US 2019/0167350 A1 · Intraoperative Systems and Methods for Determining and Providing for Display a Virtual Image Overlaid onto a Visual Image of a Bone

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0167350 A1
- Document ID: US20190167350A1
- Status: Published application
- Kind: A1
- Application number: 16/260,251
- Filed: 2019-01-29
- Published: 2019-06-06
- Earliest claimed: 2015-09-11
- Assignees: AOD Holdings, LLC
- Applicants: AOD Holdings, LLC
- Inventors: Thomas D. Ferro; Scott Gill; Austin Ferro; Donald Lee; Joe Phillips
- Web page: https://patentgrep.com/patent/US20190167350A1

## CPC

- A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations
- A61B 5/0077: {Devices for viewing the surface of the body, e.g. camera, magnifying lens}
- A61B 5/064: {using markers (A61B5/062 takes precedence)}
- A61B 5/1079: {using optical or photographic means}
- A61B 5/1127: {using markers}
- A61B 5/4504: {Bones (A61B5/4547 takes precedence)}
- A61B 5/4887: {Locating particular structures in or on the body}
- A61B 5/7425: {Displaying combinations of multiple images regardless of image source, e.g. displaying a reference anatomical image with a live image}
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 90/361: {Image-producing devices, e.g. surgical cameras}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G16H 50/50: for simulation or modelling of medical disorders
- A61B 5/11: Measuring movement of the entire body or parts thereof, e.g. head or hand tremor or mobility of a limb {(A61B5/1038 takes precedence; motion detection to correct for motion artifacts in physiological signals A61B5/721)}
- A61B 5/743: {Displaying an image simultaneously with additional graphical information, e.g. symbols, charts, function plots}
- A61B 2034/105: {Modelling of the patient, e.g. for ligaments or bones}
- A61B 2034/107: {Visualisation of planned trajectories or target regions}
- A61B 2034/108: {Computer aided selection or customisation of medical implants or cutting guides}
- A61B 2034/2055: {Optical tracking systems}
- A61B 2034/2065: {Tracking using image or pattern recognition}
- A61B 2034/252: {indicating steps of a surgical procedure}
- A61B 2090/3983: {Reference marker arrangements for use with image guided surgery}
- A61B 2505/05: Surgical care
- A61B 2576/00: Medical imaging apparatus involving image processing or analysis (A61B1/00009, A61B6/52 and A61B8/52 take precedence)
- G06T 2210/41: Medical
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

Example methods and systems may be used intraoperatively to help surgeons perform accurate and replicable surgeries, such as knee arthroplasty surgeries. An example system combines real time measurement and tracking components with functionality to compile data collected by the hardware to register a bone of a patient, calculate an axis (e.g., mechanical axis) of the leg of the patient, assist a surgeon in placing cut guides, and verify a placement of an inserted prosthesis.

## Claims

### Claim 1

A system comprising:
a measurement device to collect information of a surface of a bone of a limb of a patient, to track one or more reference points positioned on one or more bones or an anatomy of the patient, and to track one or more reference points positioned on a surgical device;
one or more processors; and
data storage including instructions executable by the one or more processors for performing functions comprising:
determining a visual model of the bone based on the collected information of the surface of the bone;
recognizing a position and a movement of the bone in physical space based on the tracked reference points positioned on the one or more of the bone or the anatomy of the patient and causing the visual model of the bone to reflect the position and the movement of the bone in physical space;
recognizing a position and a movement of the surgical device in physical space based on the tracked reference points positioned on the surgical device;
determining an axis of the limb based on reference points positioned on a medial malleoli of the patient, a lateral malleoli of the patient, and a base of an anterior cruciate ligament of the patient; and
providing for display a virtual image overlaid onto the visual model of the bone, wherein the virtual image includes information for assisting in placement of the surgical device at a position on the bone based on the position and the movement of the surgical device in physical space and includes a display of the axis of the limb.

### Claim 2 (depends on 1)

The system of claim 1, wherein the one or more processors perform the functions intraoperatively during surgery.

### Claim 3 (depends on 1)

The system of claim 1, wherein the function of providing for display the virtual image comprises:
receiving a live image of the bone;
providing for display the live image of the bone; and
displaying a virtual cutting plane onto the live image of the bone.

### Claim 4 (depends on 1)

The system of claim 1, wherein the one or more processors further perform functions comprising:
providing a virtual view of a resection on the bone.

### Claim 5 (depends on 1)

The system of claim 1, wherein the function of providing for display the virtual image further comprises:
providing for display a graphic illustrating placement of a prosthesis in relation to the bone.

### Claim 6 (depends on 1)

The system of claim 1, wherein the function of providing for display the virtual image further comprises:
providing for display information indicating one or more of a lateral gap of a knee of the patient, a medial gap of the knee of the patient, a valgus alignment of the knee of the patient, a flexion of a femoral cut, and a slope of a tibial cut.

### Claim 7 (depends on 1)

The system of claim 1, wherein the function of providing for display the virtual image comprises:
displaying the virtual image including placement of the surgical device, a graphic illustrating placement of a prosthesis in relation to the bone, and information indicating one or more of a lateral gap, a medial gap, and a valgus alignment of a knee of the patient with placement of the prosthesis.

### Claim 8 (depends on 1)

The system of claim 1, further comprising:
a guide positioning device attached to the patient and for adjusting the surgical device along one or more axes as well as along one or more of a pitch, a roll, and a yaw, and
wherein the measurement device tracks a position of the guide positioning device.

### Claim 9 (depends on 1)

The system of claim 1, further comprising:
one or more cameras for capturing a live image of the bone; and
wherein the function of providing for display the virtual image comprises further providing for display the virtual image overlaid on the live image of the bone.

### Claim 10 (depends on 1)

The system of claim 1, wherein the function of determining the axis of the limb comprises:
determining a median point between the medial malleoli of the patient and the lateral malleoli of the patient; and
determining a virtual line from the median point up to a point at the base of the anterior cruciate ligament of the patient.

### Claim 11 (depends on 1)

The system of claim 1, wherein the function of determining the axis of the limb comprises:
receiving data of an ankle of the patient;
determining peaks of sides of the ankle from at least two points based on the data of the ankle of the patient;
determining a median point between the peaks of the ankle; and
determining a virtual line from the median point up to a point at the base of the anterior cruciate ligament of the patient.

### Claim 12 (depends on 1)

The system of claim 1, wherein the function of determining the axis of the limb comprises:
receiving data of the limb of the patient;
extrapolating, based on the data of the limb of the patient, a point of rotation of a hip of the patient during movement of the limb based on tracked reference points positioned on a distal end of a femur; and
determining a virtual line from the point of rotation of the hip up to a point at the base of the anterior cruciate ligament of the patient.

### Claim 13 (depends on 1)

The system of claim 1, further comprising:
wherein the measurement device is one or more cameras for capturing live images.

### Claim 14

A method comprising:
receiving information, collected by a measurement device, of a surface of a bone of a limb of a patient;
determining a visual model of the bone based on the collected information of the surface of the bone;
recognizing a position and a movement of the bone in physical space based on tracked reference points positioned on one or more of the bone or an anatomy of the patient and causing the visual model of the bone to reflect the position and the movement of the bone in physical space;
recognizing a position and a movement of a surgical device in physical space based on tracked reference points positioned on the surgical device; and
determining an axis of the limb based on reference points positioned on a medial malleoli of the patient, a lateral malleoli of the patient, and a base of an anterior cruciate ligament of the patient; and
providing for display a virtual image overlaid onto the visual model of the bone, wherein the virtual image includes information for assisting in placement of the surgical device at a position on the bone based on the position and the movement of the surgical device in physical space and includes a display of the axis of the limb.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
receiving the information of the surface of the bone as exposed from the limb of the patient in real time during surgery.

### Claim 16 (depends on 14)

The method of claim 14, wherein receiving the information, collected by the measurement device, of the surface of the bone of the limb of the patient comprises:
receiving the information using a scanning tool in real time during surgery.

### Claim 17 (depends on 14)

The method of claim 14, further comprising:
providing a virtual view of a resection on the bone.

### Claim 18 (depends on 14)

The method of claim 14, further comprising:
displaying the virtual image including placement of the surgical device, a graphic illustrating placement of a prosthesis in relation to the bone, and information indicating one or more of a lateral gap, a medial gap, and a valgus alignment of a knee of the patient with placement of the prosthesis.

### Claim 19

A non-transitory computer-readable medium having stored thereon instructions that when executed by a computing device that includes one or more processors causes the computing device to perform functions comprising:
receiving information, collected by a measurement device, of a surface of a bone of a limb of a patient;
determining a visual model of the bone based on the collected information of the surface of the bone;
recognizing a position and a movement of the bone in physical space based on tracked reference points positioned on one or more of the bone or an anatomy of the patient and causing the visual model of the bone to reflect the position and the movement of the bone in physical space;
recognizing a position and a movement of a surgical device in physical space based on tracked reference points positioned on the surgical device; and
determining an axis of the limb based on reference points positioned on a medial malleoli of the patient, a lateral malleoli of the patient, and a base of an anterior cruciate ligament of the patient; and
providing for display a virtual image overlaid onto the visual model of the bone, wherein the virtual image includes information for assisting in placement of the surgical device at a position on the bone based on the position and the movement of the surgical device in physical space and includes a display of the axis of the limb.

### Claim 20 (depends on 19)

The non-transitory computer-readable medium of claim 19, wherein the function of providing for display the virtual image comprises:
receiving a live image of the bone;
providing for display the live image of the bone; and
displaying a virtual cutting plane onto the live image of the bone.

### Claim 21 (depends on 19)

The non-transitory computer-readable medium of claim 19, wherein the function of providing for display the virtual image comprises:
receiving the information of the bone as exposed from the limb of the patient in real time during surgery.

### Claim 22 (depends on 19)

The non-transitory computer-readable medium of claim 19, wherein the function of receiving the information, collected by the measurement device, of the surface of the bone of the limb of the patient comprises:
receiving the information using a scanning tool in real time during surgery.

### Claim 23 (depends on 19)

The non-transitory computer-readable medium of claim 19, wherein the function of providing for display the virtual image comprises:
providing a virtual view of a resection on the bone.

### Claim 24 (depends on 19)

The non-transitory computer-readable medium of claim 19, wherein the function of providing for display the virtual image comprises:
displaying the virtual image including placement of the surgical device, a graphic illustrating placement of a prosthesis in relation to the bone, and information indicating one or more of a lateral gap, a medial gap, and a valgus alignment of a knee of the patient with placement of the prosthesis.
