# US 2020/0286404 A1 · System and method for three-dimensional augmented reality guidance for use of equipment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0286404 A1
- Document ID: US20200286404A1
- Status: Published application
- Kind: A1
- Application number: 16/727,832
- Filed: 2019-12-26
- Published: 2020-09-10
- Earliest claimed: 2017-01-24
- Assignees: Tienovix, LLC
- Applicants: Tienovix, LLC
- Inventors: William R. BURAS; Craig S. RUSSELL; Kyle Q. NGUYEN
- Web page: https://patentgrep.com/patent/US20200286404A1

## CPC

- G09B 23/286: {for scanning or photography techniques, e.g. X-rays, ultrasonics}
- A61B 8/06: Measuring blood flow
- A61B 8/085: {for locating body or organic structures, e.g. tumours, calculi, blood vessels, nodules}
- A61B 8/4245: {involving determining the position of the probe, e.g. with respect to an external reference frame or to the patient}
- A61B 8/4254: {using sensors mounted on the probe}
- A61B 8/4263: {using sensors not mounted on the probe, e.g. mounted on an external reference frame}
- A61B 8/466: {adapted to display 3D data}
- A61B 8/467: {characterised by special input means}
- A61B 8/52: {Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves}
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 90/361: {Image-producing devices, e.g. surgical cameras}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06N 20/00: Machine learning
- G09B 5/065: {Combinations of audio and video presentations, e.g. videotapes, videodiscs, television systems}
- G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS
- G16H 40/63: for local operation
- A61B 2034/2048: {using an accelerometer or inertia sensor}
- A61B 2034/2055: {Optical tracking systems}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/378: {using ultrasound}
- A61B 2090/3945: {Active visible markers, e.g. light emitting diodes}
- A61B 2562/0219: Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
- G02B 27/017: {Head mounted}
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS

## Abstract

A guidance system providing real-time, three-dimensional (3D) augmented reality (AR) feedback guidance to a user of an equipment system, to achieve improved procedural outcomes in the use of the equipment system.

## Claims

### Claim 1

A guidance system for providing real-time, three-dimensional (3D) augmented reality (AR) feedback guidance to a user of an equipment system, the guidance system comprising:
an equipment interface to an equipment system, wherein said equipment interface is capable of receiving data from the equipment system during an equipment procedure performed by a user of the equipment system;
an augmented reality user interface (ARUI) for presenting data pertaining to both real and virtual objects to the user during at least a portion of the performance of the equipment procedure;
a three-dimensional guidance system (3DGS) that is capable of sensing real-time, three-dimensional user positioning data relating to one or more of the movement, position, and orientation of at least a portion of the equipment system during said equipment procedure performed by the user;
a library containing 1) stored reference positioning data relating to one or more of the movement, position, and orientation of at least a portion of the equipment system during a reference equipment procedure and 2) stored reference outcome data relating to an outcome of said reference equipment procedure; and
a machine learning module (MLM) for providing at least one of 1) real-time position-based 3D AR feedback to the user based on the sensed user positioning data and the reference positioning data, and 2) real-time outcome-based 3D AR feedback to the user based on data received from the equipment system during the equipment procedure performed by the user and the reference outcome data.

### Claim 2 (depends on 1)

The guidance system of claim 1, further comprising a computer system containing said equipment interface, wherein the equipment system comprises one of a heavy machinery system, a precision instrument, and a precision tool coupled to the computer system via the equipment interface.

### Claim 3 (depends on 1)

The guidance system of claim 1, wherein the MLM provides the at least one of real-time position-based feedback and real-time outcome-based feedback to a user performing a task selected from a delicate task and a task that is technique-sensitive.

### Claim 4 (depends on 3)

The guidance system of claim 3, wherein the MLM provides the at least one of real-time position-based feedback and real-time outcome-based feedback to a user performing an equipment procedure selected from an item on a daily checklist, a general equipment maintenance procedure, an assembly procedure, and a testing procedure.

### Claim 5 (depends on 1)

The guidance system of claim 1 further comprising a computer system comprising said equipment interface to the equipment system, wherein said computer system is coupled to one or more of the ARUI, the 3DGS, the library, and the MLM.

### Claim 6 (depends on 1)

The guidance system of claim 1, wherein the 3DGS comprises one or more of a magnetic GPS system, a digital camera tracking system, an infrared camera system, an accelerometer, and a gyroscope.

### Claim 7 (depends on 1)

The guidance system of claim 1, wherein the 3DGS senses real-time user positioning data by sensing at least one of:
a magnetic field generated by said at least a portion of the equipment system;
the movement of one or more passive visual markers coupled to one or more of the user and a portion of the equipment system; and
the movement of one or more active visual markers coupled to one or more of the user and a portion of the equipment system.

### Claim 8 (depends on 1)

The guidance system of claim 1, wherein the library further includes instructions for performing one or more procedures using the equipment system.

### Claim 9 (depends on 1)

The guidance system of claim 1, wherein at least one of said real-time position-based 3D AR feedback and said real-time outcome-based 3D AR feedback is selected from a virtual still image, a virtual video image, sounds, or tactile information.

### Claim 10 (depends on 1)

The guidance system of claim 1, wherein at least one of said real-time position-based 3D AR feedback and said real-time outcome-based 3D AR feedback is selected from:
a virtual prompt indicating a movement correction to be performed by a user;
a virtual image or video instructing the user to change the orientation of the at least a portion of the equipment system to match a desired orientation;
a virtual image or video of a correct motion path to be taken by the user in performing the equipment procedure;
a color-coded image or video indicating one or more correct portions of the user's motion in performing the equipment procedure;
a color-coded image or video indicating one or more incorrect portions of the user's motion in performing the equipment procedure; and
an auditory instruction, virtual image, or virtual video indicating a direction for the user to move the at least a portion of the equipment system.

### Claim 11 (depends on 1)

The guidance system of claim 1, wherein the MLM provides both of said real-time position-based 3D AR feedback and said real-time outcome-based 3D AR feedback to the user of the equipment system.

### Claim 12 (depends on 1)

The guidance system of claim 1 further comprising:
a computer system containing said interface to the equipment system, and wherein said ARUI comprises a software interface located in said computer (700); and
a head mounted display (HMD) coupled to said ARUI.

### Claim 13 (depends on 1)

The guidance system of claim 1 further comprising:
a computer system, wherein said ARUI comprises a software interface located in said computer and a head mounted display (HMD) coupled to said software interface.

### Claim 14 (depends on 1)

The guidance system of claim 1, wherein the three-dimensional guidance system (3DGS) is capable of sensing real-time user positioning data relating to at least a portion of the equipment system within a volume of the user's environment during said equipment procedure performed by the user.

### Claim 15

A guidance system for providing real-time, three-dimensional (3D) augmented reality (AR) feedback guidance in the use of an equipment system, the guidance system comprising:
a computer comprising:
an equipment interface to an equipment system, wherein said equipment interface is adapted to receive data from the equipment system during an equipment procedure performed by a user of the equipment to achieve an equipment procedure outcome;
an AR user interface (ARUI) to an AR head mounted display (HMD) for presenting information pertaining to both real and virtual objects to the user during the performance of the equipment procedure;
a guidance system interface (GSI) to a three-dimensional guidance system (3DGS) that senses real-time, three-dimensional user positioning data relating to one or more of the movement, position, and orientation of at least a portion of the equipment system within a volume of a user's environment during the equipment procedure performed by the user;
a library containing 1) stored reference positioning data relating to one or more of the movement, position, and orientation of at least a portion of the equipment system during a reference performance of the equipment procedure and 2) stored reference outcome data relating to an outcome of a reference performance of the equipment procedure; and
a machine learning module (MLM) for providing at least one of 1) real-time position-based 3D AR feedback to the user based on the sensed user positioning data, and 2) real-time outcome-based 3D AR feedback to the user based on the equipment procedure outcome, the MLM comprising:
a position-based feedback module comprising:
a first module for receiving and analyzing the real-time, three-dimensional user positioning data;
a second module for comparing the real-time, three-dimensional user positioning data to the stored reference positioning data and generating an output; and
a third module for generating real-time position-based 3D AR feedback based on the output of the second module, and providing said real-time position-based 3D AR feedback to the user via the ARUI; and
an outcome-based feedback module comprising:
a fourth module for receiving real-time data from the equipment system via said equipment interface as the user performs the equipment procedure, wherein the real-time data is related to an outcome of the equipment procedure;
a fifth module for comparing the real-time data related to an outcome of the equipment procedure received from the fourth module to the stored reference outcome data, and for generating an output; and
a sixth module for generating real-time outcome-based 3D AR feedback based on the output of the fifth module, and providing said real-time outcome-based 3D AR feedback to the user via the ARUI.

### Claim 16 (depends on 15)

The guidance system of claim 15, further comprising the AR HMD.

### Claim 17 (depends on 15)

The guidance system of claim 15, wherein at least one of the fifth module and the sixth module comprises a neural network.

### Claim 18 (depends on 17)

The guidance system of claim 17, wherein the neural network is a convolutional neural network.

### Claim 19 (depends on 15)

The guidance system of claim 15, wherein said third module provides said real-time position-based 3D AR feedback to the user visually in a HMD relative to at least a portion of the equipment system, and said sixth module provides said real-time outcome-based 3D AR feedback to the user visually in a HMD relative to at least a portion of the equipment system.

### Claim 20

A guidance system for providing real-time, three-dimensional (3D) augmented reality (AR) feedback guidance to a user of an equipment system, the guidance system comprising:
an equipment interface to an equipment system, wherein said equipment interface is capable of receiving data from the equipment system during an equipment procedure performed by a user of the equipment system;
an augmented reality user interface (ARUI) for presenting data pertaining to both real and virtual objects to the user during at least a portion of the performance of the equipment procedure by the user;
a three-dimensional guidance system (3DGS) that is capable of sensing real-time, three-dimensional user positioning data relating to one or more of the movement, position, and orientation of at least a portion of the equipment system during said equipment procedure performed by the user;
a library containing 1) stored reference positioning data relating to one or more of the movement, position, and orientation of at least a portion of the equipment system during a reference performance of the equipment procedure and 2) stored reference outcome data relating to an outcome of said reference equipment procedure; and
a machine learning module (MLM) for providing 1) real-time position-based 3D AR feedback to the user based on the sensed user positioning data and the reference positioning data, and 2) real-time outcome-based 3D AR feedback to the user based on data received from the equipment system during the equipment procedure performed by the user and the reference outcome data, the MLM comprising:
a position-based feedback module comprising:
a first module for receiving and analyzing the real-time, three-dimensional user positioning data sensed by the 3DGS during the equipment procedure performed by the user;
a second module for comparing the real-time, three-dimensional user positioning data to the stored reference positioning data and generating an output; and
a third module for generating real-time position-based 3D AR feedback based on the output of the second module, and providing said real-time position-based 3D AR feedback to the user via the ARUI; and
an outcome-based feedback module comprising:
a fourth module for receiving real-time data from the equipment system via said equipment interface during the equipment procedure performed by the user, wherein the real-time data is related to an outcome of the equipment procedure;
a fifth module for comparing the real-time data related to an outcome of the equipment procedure received from the fourth module to the stored reference outcome data, and for generating an output; and
a sixth module for generating real-time outcome-based 3D AR feedback based on the output of the fifth module, and providing said real-time outcome-based 3D AR feedback to the user via the ARUI.