# US 10,120,413 B2 · System and method for enhanced training using a virtual reality environment and bio-signal data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,120,413 B2
- Document ID: US10120413B2
- Status: Granted
- Kind: B2
- Application number: 14/851,853
- Filed: 2015-09-11
- Published: 2016-03-17
- Granted: 2018-11-06
- Earliest claimed: 2014-09-11
- Assignees: Interaxon Inc.
- Applicants: Interaxon Inc.
- Inventors: Christopher Allen Aimone; Trevor Coleman; Ariel Stephanie Garten; Kapil Jay Mishra Vidyarthi; Locillo (Lou) Giuseppe Pino; Michael Apollo Chabior; Paul Harrison Baranowski; Raul Rajiv Rupsingh; Madeline Ashby; Paul V. Tadich
- Web page: https://patentgrep.com/patent/US10120413B2

## CPC

- G06F 1/163: {Wearable computers, e.g. on a belt}
- A61B 3/113: for determining or recording eye movement
- A61B 5/0006: {ECG or EEG signals}
- A61B 5/0022: {Monitoring a patient using a global network, e.g. telephone networks, internet}
- A61B 5/0024: {for multiple sensor units attached to the patient, e.g. using a body or personal area network}
- A61B 5/1114: {Tracking parts of the body}
- A61B 5/16: Devices for psychotechnics (using teaching or educational appliances G09B1/00 - G09B7/00); Testing reaction times {; Devices for evaluating the psychological state}
- A61B 5/165: {Evaluating the state of mind, e.g. depression, anxiety}
- A61B 5/372: Analysis of electroencephalograms
- A61B 5/378: Visual stimuli
- A61B 5/38: Acoustic or auditory stimuli
- A61B 5/384: Recording apparatus or displays specially adapted therefor
- A61B 5/486: {Biofeedback (using electroencephalography [EEG] A61B5/375)}
- A61B 5/6803: {Head-worn items, e.g. helmets, masks, headphones or goggles}
- A61B 5/6814: {Head}
- A61B 5/6821: {Eye}
- A61B 5/6831: {Straps, bands or harnesses}
- A61B 5/7203: {for noise prevention, reduction or removal}
- A61B 5/7264: {Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems}
- A61B 5/744: {Displaying an avatar, e.g. an animated cartoon character}
- A61B 5/7445: {Display arrangements, e.g. multiple display units}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G16H 40/67: for remote operation
- A61B 2503/12: Healthy persons not otherwise provided for, e.g. subjects of a marketing survey
- A61B 2560/0475: Special features of memory means, e.g. removable memory cards
- A61B 2560/0493: controlled by voice
- A61B 2562/0219: Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
- A61B 2562/125: characterised by the manufacture of electrodes
- A61B 2562/164: the sensor is mounted in or on a conformable substrate or carrier
- A61B 2562/166: the sensor is mounted on a specially adapted printed circuit board
- G06F 2203/011: Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

A training apparatus has an input device and a wearable computing device with a bio-signal sensor and a display to provide an interactive virtual reality (“VR”) environment for a user. The bio-signal sensor receives bio-signal data from the user. The user interacts with content that is presented in the VR environment. The user interactions and bio-signal data are scored with a user state score and a performance scored. Feedback is given to the user based on the scores in furtherance of training. The feedback may update the VR environment and may trigger additional VR events to continue training.

## Claims

### Claim 1

A training apparatus comprising:
an input device and a wearable computing device with a bio-signal sensor and a display to provide an interactive continuous virtual reality (“VR”) environment for a user, the VR environment containing virtual elements, the bio-signal sensor receives bio-signal data from the user, the bio-signal sensor comprising a brainwave sensor;
the computing device having or in communication with a processor configured to:
as part of the interactive continuous VR environment, present content on the display where the content includes the virtual elements and has a VR event occurring within the interactive continuous VR environment, the VR event having one or more changes on the virtual elements in the VR environment, the VR event having desired user states, and desired effects;
continuously receive user manual inputs from the input device for user interaction with the virtual elements in the interactive continuous VR environment including during the VR event;
continuously receive the bio-signal data of the user from the bio-signal sensor during the VR event;
process the bio-signal data to determine user states of the user, including brain states, during the VR event, the user states are processed using a user profile stored in a data storage device accessible by the processor and the user states include brain states;
determine a user state score by comparing the user states of the user to the desired user states during the course of the VR event;
determine a performance score by comparing the effects to the desired effects during the course of the VR event; and
provide feedback to the user wherein the feedback is based on a combination of the user state score and the performance score,
wherein the user manual inputs are continuously received independent of the user state score.

### Claim 2 (depends on 1)

The apparatus of claim 1 where the wearable computing device further comprises an inertial sensor and the bio-signal sensor further comprises a facial bio-signal sensor, wherein the facial sensor includes an EOG sensor, and where the bio-signal data further comprises data from the facial bio-signal sensor and the inertial sensor; the computing device is further configured to:
receive the bio-signal data from the EOG sensor and the inertial sensor for a user head and eye gaze direction
wherein the user states further comprises the user head and eye gaze direction, and the desired user states further comprises a desired user head and eye gaze direction.

### Claim 3 (depends on 1)

The apparatus of claim 1 wherein the brain states comprises one or more of ability of operator to learn; prediction error; and emotional state leading to impaired thinking.

### Claim 4 (depends on 1)

The apparatus of claim 1 further comprising presenting the user state score and the performance score synchronized with the content and the VR event to assist the user to better attain the desired user states and desired manual inputs on the input device.

### Claim 5 (depends on 4)

The apparatus of claim 4 where
the computing device is further configured to:
revise the content in response to the feedback provided to the user where the user is further trained on the revised content.

### Claim 6 (depends on 4)

The apparatus of claim 4 wherein the user state score further comprises failure brain states.

### Claim 7 (depends on 1)

The apparatus of claim 1 where the display is a stereoscopic display.

### Claim 8 (depends on 1)

The apparatus of claim 1 further comprising
a second display for presenting the content and a visual representation of the user states and manual inputs of the user in real time.

### Claim 9 (depends on 1)

The apparatus of claim 1 wherein the computing device having or in communication with the processor is further configured to:
provide real time or near real time feedback to the user during the presentation of the content.

### Claim 10 (depends on 1)

The apparatus of claim 1, wherein the VR event is associated with event time data and a portion of bio-signal data is associated with bio-signal time data corresponding to the event time data, wherein the processor is configured to identify a portion of the bio-signal data based on the event time data and process the portion of the bio-signal data to determine the user states during the VR event, the bio-signal time data synchronized to the event time data.

### Claim 11 (depends on 1)

The apparatus of claim 1, wherein the VR event is associated with event time data and the bio-signal data is associated with bio-signal time data, and wherein the processor is configured to identify a time interval based on an expected response time for the VR event and the event time data, identify a portion of the bio-signal data based on the time interval and the bio-signal time data, and process the portion of the bio-signal data to determine the user states during the VR event, the bio-signal time data synchronized to the event time data.

### Claim 12

A training method implemented using an input device and a wearable computing device having or in communication with a processor, a bio-signal sensor and a display to provide an interactive continuous virtual reality (“VR”) environment for a user, the VR environment containing a plurality of virtual elements, the bio-signal sensor receives bio-signal data from the user, the bio-signal sensor comprising a brainwave sensor; the training method comprising:
as part of the interactive continuous VR environment, presenting content on the display where the content has a VR event occurring within the interactive continuous VR environment, the VR event having one or more changes on at least a portion of the plurality of virtual elements in the VR environment, the VR event having desired user states, and desired effects;
continuously receiving user manual inputs from the input device for user interaction with the virtual elements in the interactive continuous VR environment including during the VR event;
continuously receiving the bio-signal data of the user from the bio-signal sensor during the VR event;
processing the bio-signal data to determine user states of the user, including brain states, during the VR event, the user states are processed using a user profile stored in a data storage device accessible by the processor and the user states include brain states;
determining a user state score by comparing the user states of the user to the desired user states during the course of the VR event;
determining a performance score by comparing the effects to the desired effects during the course of the VR event; and
providing feedback to the user wherein the feedback is based on a combination of the user state score and the performance score,
wherein the user manual inputs are continuously received independent of the user state score.

### Claim 13 (depends on 12)

The training method of claim 12, where the wearable computing device further comprises an inertial sensor and the bio-signal sensor further comprises a facial bio-signal sensor, wherein the facial sensor includes an EOG sensor, and where the bio-signal data further comprises data from the facial bio-signal sensor and the inertial sensor; the method further comprising
receiving the bio-signal data from the EOG sensor and the inertial sensor for a user head and eye gaze direction;
wherein the user states further comprises the user head and eye gaze direction, and the desired user states further comprises a desired user head and eye gaze direction.

### Claim 14 (depends on 12)

The training method of claim 12, wherein the brain states comprises one or more of ability of operator to learn; prediction error; and emotional state leading to impaired thinking.

### Claim 15 (depends on 12)

The training method of claim 12, further comprising post presenting the user state score and the performance score synchronized with the content and the VR event to assist the user to better attain the desired user states and desired manual inputs on the input device.

### Claim 16 (depends on 15)

The training method of claim 15, further comprising
revising the content in response to the feedback provided to the user where the user is further trained on the revised content.

### Claim 17 (depends on 15)

The training method of claim 15, wherein the user state score further comprises failure brain states.

### Claim 18 (depends on 12)

The training method of claim 12, where the display is a stereoscopic display.

### Claim 19 (depends on 15)

The training method of claim 15, further comprising
presenting the content and the user states and manual inputs of the user in real time on a second display.

### Claim 20 (depends on 15)

The training method of claim 15, further comprising
providing real time feedback to the user during the presentation of the content.

### Claim 21 (depends on 1)

The apparatus of claim 1, wherein the desired user state includes a distribution of desired brain states and wherein the determining of the user state score includes determining the deviation of the brain state from the distribution of desired brain states.

### Claim 22 (depends on 1)

The apparatus of claim 1, wherein the processor is further configured to dynamically update the user profile using the bio-signal data.