# US 10,768,665 B2 · System and method for enhanced training using a virtual reality environment and bio-signal data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,768,665 B2
- Document ID: US10768665B2
- Status: Granted
- Kind: B2
- Application number: 16/144,854
- Filed: 2018-09-27
- Published: 2019-01-31
- Granted: 2020-09-08
- Earliest claimed: 2014-09-11
- Assignees: Interaxon Inc.
- Applicants: Interaxon Inc.
- Inventors: Christopher Allen Aimone; Trevor C E Coleman; Ariel Stephanie Garten; Kapil Jay Mishra Vidyarthi; Locillo (Lou) Giuseppe Pino; Michael Apollo Chabior; Paul Harrison Baranowski; Raul Rajiv Rupsingh; Madeline Ashby; Paul V Tadich
- Web page: https://patentgrep.com/patent/US10768665B2

## CPC

- G06F 1/163: {Wearable computers, e.g. on a belt}
- A61B 3/113: for determining or recording eye movement
- A61B 5/0006: {ECG or EEG signals}
- A61B 5/0022: {Monitoring a patient using a global network, e.g. telephone networks, internet}
- A61B 5/0024: {for multiple sensor units attached to the patient, e.g. using a body or personal area network}
- A61B 5/1114: {Tracking parts of the body}
- A61B 5/16: Devices for psychotechnics (using teaching or educational appliances G09B1/00 - G09B7/00); Testing reaction times {; Devices for evaluating the psychological state}
- A61B 5/165: {Evaluating the state of mind, e.g. depression, anxiety}
- A61B 5/372: Analysis of electroencephalograms
- A61B 5/378: Visual stimuli
- A61B 5/38: Acoustic or auditory stimuli
- A61B 5/384: Recording apparatus or displays specially adapted therefor
- A61B 5/486: {Biofeedback (using electroencephalography [EEG] A61B5/375)}
- A61B 5/6803: {Head-worn items, e.g. helmets, masks, headphones or goggles}
- A61B 5/6814: {Head}
- A61B 5/6821: {Eye}
- A61B 5/6831: {Straps, bands or harnesses}
- A61B 5/7203: {for noise prevention, reduction or removal}
- A61B 5/7264: {Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems}
- A61B 5/744: {Displaying an avatar, e.g. an animated cartoon character}
- A61B 5/7445: {Display arrangements, e.g. multiple display units}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G16H 40/67: for remote operation
- A61B 2503/12: Healthy persons not otherwise provided for, e.g. subjects of a marketing survey
- A61B 2560/0475: Special features of memory means, e.g. removable memory cards
- A61B 2560/0493: controlled by voice
- A61B 2562/0219: Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
- A61B 2562/125: characterised by the manufacture of electrodes
- A61B 2562/164: the sensor is mounted in or on a conformable substrate or carrier
- A61B 2562/166: the sensor is mounted on a specially adapted printed circuit board
- G06F 2203/011: Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

A training apparatus has an input device and a wearable computing device with a bio-signal sensor and a display to provide an interactive virtual reality (“VR”) environment for a user. The bio-signal sensor receives bio-signal data from the user. The user interacts with content that is presented in the VR environment. The user interactions and bio-signal data are scored with a user state score and a performance scored. Feedback is given to the user based on the scores in furtherance of training. The feedback may update the VR environment and may trigger additional VR events to continue training.

## Claims

### Claim 1

A training apparatus comprising:
an input device and a wearable computing device with a bio-signal sensor, a sensor for measuring movement and a display to provide an interactive continuous virtual reality (“VR”) environment for a user, the VR environment containing virtual elements, the bio-signal sensor receives bio-signal data from the user, the bio-signal sensor comprising a brainwave sensor;
the computing device having or in communication with a processor configured to:
as part of the interactive continuous VR environment, present content on the display where the content includes the virtual elements and has a VR event occurring within the interactive continuous VR environment, the VR event having one or more changes on the virtual elements in the VR environment, the VR event having desired user states;
continuously receive user manual inputs from the input device for user interaction with the virtual elements in the interactive continuous VR environment including during the VR event;
continuously receive the bio-signal data of the user from the bio-signal sensor and the sensor for measuring movement, including during the VR event;
process the bio-signal data to determine user states of the user, including brain states and a user head and eye gaze direction, during the VR event;
determine a user state score by comparing the user states of the user to the desired user states during the course of the VR event; and
modify the interactive continuous VR environment based on the user state score
wherein the user manual inputs are continuously received independent of the user state score.

### Claim 2 (depends on 1)

The apparatus of claim 1 wherein the user states are processed using a user profile stored in a data storage device accessible by the processor.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein the processor is further configured to dynamically update the user profile using the bio-signal data.

### Claim 4 (depends on 1)

The apparatus of claim 1 wherein the processor is further configured to provide feedback to the user wherein the feedback is based on the user state score.

### Claim 5 (depends on 1)

The apparatus of claim 1 wherein the brain states comprises one or more of ability of operator to learn; prediction error; and emotional state leading to impaired thinking.

### Claim 6 (depends on 1)

The apparatus of claim 1 where
the computing device is further configured to:
revise the content based on the user state score where the user is further trained on the revised content.

### Claim 7 (depends on 1)

The apparatus of claim 1 further comprising
a second display for presenting the content and a visual representation of the user states and manual inputs of the user in real time.

### Claim 8 (depends on 1)

The apparatus of claim 1 wherein the computing device having or in communication with the processor is further configured to:
provide real time or near real time feedback to the user during the presentation of the content.

### Claim 9 (depends on 1)

The apparatus of claim 1, wherein the VR event is associated with event time data and a portion of bio-signal data is associated with bio-signal time data corresponding to the event time data, wherein the processor is configured to identify a portion of the bio-signal data based on the event time data and process the portion of the bio-signal data to determine the user states during the VR event, the bio-signal time data synchronized to the event time data.

### Claim 10 (depends on 1)

The apparatus of claim 1, wherein the VR event is associated with event time data and the bio-signal data is associated with bio-signal time data, and wherein the processor is configured to identify a time interval based on an expected response time for the VR event and the event time data, identify a portion of the bio-signal data based on the time interval and the bio-signal time data, and process the portion of the bio-signal data to determine the user states during the VR event, the bio-signal time data synchronized to the event time data.

### Claim 11 (depends on 1)

The apparatus of claim 1, wherein the desired user state includes a distribution of desired brain states and wherein the determining of the user state score includes determining the deviation of the brain state from the distribution of desired brain states.

### Claim 12

A training method implemented using an input device and a wearable computing device having or in communication with a processor, a bio-signal sensor, a sensor for measuring movement and a display to provide an interactive continuous virtual reality (“VR”) environment for a user, the VR environment containing a plurality of virtual elements, the bio-signal sensor receives bio-signal data from the user, the bio-signal sensor comprising a brainwave sensor; the training method comprising:
as part of the interactive continuous VR environment, presenting content on the display where the content has a VR event occurring within the interactive continuous VR environment, the VR event having one or more changes on at least a portion of the plurality of virtual elements in the VR environment, the VR event having desired user states;
continuously receiving user manual inputs from the input device for user interaction with the virtual elements in the interactive continuous VR environment including during the VR event;
continuously receiving the bio-signal data of the user from the bio-signal sensor and the sensor for measuring movement, including during the VR event;
processing the bio-signal data to determine user states of the user, including brain states and a user head and eye gaze direction, during the VR event;
determining a user state score by comparing the user states of the user to the desired user states during the course of the VR event; and
modify the interactive continuous VR environment based on the user state score,
wherein the user manual inputs are continuously received independent of the user state score.

### Claim 13 (depends on 12)

The training method of claim 12 wherein the user states are processed using a user profile stored in a data storage device accessible by the processor.

### Claim 14 (depends on 12)

The training method of claim 12 further comprising: providing feedback to the user wherein the feedback is based on the user state score.

### Claim 15 (depends on 12)

The training method of claim 12, wherein the brain states comprises one or more of ability of operator to learn; prediction error; and emotional state leading to impaired thinking.

### Claim 16 (depends on 12)

The training method of claim 12, further comprising
revising the content based on the user state score where the user is further trained on the revised content.

### Claim 17 (depends on 12)

The training method of claim 12, further comprising
presenting the content and the user states and manual inputs of the user in real time on a second display.

### Claim 18 (depends on 12)

The training method of claim 12, further comprising
providing real time feedback to the user during the presentation of the content.

### Claim 19 (depends on 1)

The apparatus of claim 1, wherein the desired user states include a desired user head and eye gaze direction.

### Claim 20 (depends on 12)

The training method of claim 12, wherein the desired user states include a desired user head and eye gaze direction.