# US 12,002,180 B2 · Immersive ecosystem

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,180 B2
- Document ID: US12002180B2
- Status: Granted
- Kind: B2
- Application number: 17/849,123
- Filed: 2022-06-24
- Published: 2022-10-13
- Granted: 2024-06-04
- Earliest claimed: 2019-08-05
- Applicants: Brightline Interactive, LLC
- Inventors: Tyler H. Gates
- Web page: https://patentgrep.com/patent/US12002180B2

## CPC

- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G09B 5/02: with visual presentation of the material to be studied, e.g. using film strip
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 27/017: {Head mounted}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06N 20/00: Machine learning
- G09B 9/003: {for military purposes and tactics}
- G09B 19/00: Teaching not covered by other main groups of this subclass (teaching or practice apparatus for gun-aiming or gun-laying F41G3/26)
- G09B 23/28: for medicine
- G16H 20/70: relating to mental therapies, e.g. psychological therapy or autogenous training
- G16H 40/63: for local operation
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G16H 50/30: for calculating health indices; for individual health risk assessment
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- A61B 2034/101: {Computer-aided simulation of surgical operations}
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)

## Abstract

An immersive training system, and method performed with such system, which comprises one or more sensors configured to measure at least one response of a user to a three-dimensional (3D) asset of a 3D rendering that is being rendered on a virtual reality (VR) display, the 3D rendering comprising a plurality of 3D assets. The immersive training system and method compares, in real time, the received measured response of the user to a corresponding predetermined user response value to the 3D asset and, based on the user response comparison and a generated user-customized dynamic decision matrix, modifies at least one of the plurality of 3D assets of the 3D rendering. A customized dynamic decision matrix is generated for each user based on, for example, a corresponding personal performance profile and a neural efficiency score.

## Claims

### Claim 1

An immersive training system comprising:
one or more sensors configured to measure at least one response of a user to a three-dimensional (3D) asset of a 3D rendering that is being rendered on a virtual reality (VR) display, the 3D rendering comprising a plurality of 3D assets;
a processor;
a non-transitory computer-readable medium; and
a set of instructions stored on the computer-readable medium that is operable when executed on the processor to:
generate a personal performance profile specific to the user and based on prior stored measured user responses to one or more 3D assets of the plurality of 3D assets of the 3D rendering;
receive, in real-time, the at least one measured response of the user to the 3D asset,
update the user-specific personal performance profile based on the received at least one measured response of the user to the 3D asset;
compare, in real-time, the received at least one measured response of the user at a given time with a corresponding predetermined user response value to the 3D asset at a corresponding predetermined time value to the 3D asset;
generate a dynamic decision matrix customized for the user based on the user-specific personal performance profile and a neural efficiency score of the user, and
modify, in real-time, at least one of the plurality of 3D assets of the 3D rendering based on the dynamic decision matrix customized for the user and the user response comparison.

### Claim 2 (depends on 1)

The immersive training system of claim 1, wherein the one or more sensors comprises:
an active sensor configured to measure a physical response of the user to the 3D asset; or
a cognitive or physiological sensor configured to measure a passive cognitive or passive physiological response of the user to the 3D asset.

### Claim 3 (depends on 1)

The immersive training system of claim 1, wherein the one or more sensors comprises:
an active sensor configured to measure a physical response of the user to the 3D asset; and
a cognitive or physiological sensor configured to measure a passive cognitive or passive physiological response of the user to the 3D asset, and
wherein the set of instructions stored on the computer-readable medium being further operable when executed on the processor to generate the neural efficiency score of the user based on:
a proficiency score determined, in real-time, based on a comparison between the measured physical response of the user to the 3D asset and a corresponding measurement of correctness, and the measured passive cognitive or passive physiological response of the user to the 3D asset.

### Claim 4 (depends on 1)

The immersive training system of claim 1, wherein the corresponding predetermined user response value to the 3D asset is determined, in part, based on an experience level of the user, a training level of the user, or the neural efficiency score of the user.

### Claim 5 (depends on 3)

The immersive training system of claim 3, wherein the set of instructions stored on the computer-readable medium being further operable when executed on the processor to modify, in real-time, the at least one of the plurality of 3D assets of the 3D rendering based on a comparison between the generated user neural efficiency score and a neural efficiency threshold.

### Claim 6 (depends on 1)

The immersive training system of claim 1, wherein the generated user-specific personal performance profile is further based on one or more of: demographic information of the user, addiction history of the user, an emotional and personality profile of the user, program objectives for the 3D rendering, and feedback from a trainer observing the user's performance.

### Claim 7 (depends on 3)

The immersive training system of claim 3, wherein the at least one of the plurality of 3D assets of the 3D rendering is further modified to maintain the measured passive cognitive or passive physiological response of the use rat a predetermined peak performance stress level.

### Claim 8 (depends on 2)

The immersive training system of claim 2, wherein the cognitive or physiological sensor comprises one or more of: functional near-infrared spectroscopy (fNIRS) technology, electroencephalogram (EEG) technology, an electrocardiogram (ECG) sensor, a heart rate sensor, and galvanic skin response (GSR) technology, or wherein the active sensor comprises one of: a motion capture (Mocap) sensors, an eye tracker sensor, and a body pressure sensor.

### Claim 9 (depends on 1)

The immersive training system of claim 1, wherein the 3D rendering that is being rendered on the VR display is during a VR scenario, and wherein the set of instructions stored on the computer-readable medium is further operable when executed on the processor to:
receive the at least one measured response of the user during the VR scenario,
compare, during the VR scenario, the received at least one measured response of the user at a given time with the corresponding predetermined user response value to the 3D asset at a corresponding predetermined time value of the VR scenario; and
modify the at least one of the plurality of 3D assets of the 3D rendering during the VR scenario.

### Claim 10 (depends on 9)

The immersive training system of claim 9, wherein the corresponding predetermined user response value is pre-programmed in a cloud-based database, and the corresponding predetermined time value is pre-programmed on the computer-readable medium.

### Claim 11 (depends on 9)

The immersive training system of claim 9, wherein the corresponding predetermined user response value, and the corresponding predetermined time value, are pre-programmed in a cloud-based database.

### Claim 12

A computer-implemented method for immersive training comprising:
measuring, with one or more sensors, at least one response of a user to a three-dimensional (3D) asset of a 3D rendering that is being rendered on a virtual reality (VR) display, the 3D rendering comprising a plurality of 3D assets; and
executing, on a processor, a set of instructions stored on a computer readable medium that is operable to:
generate a personal performance profile specific to the user and based on prior stored measured user responses to one or more of the 3D assets of the plurality of assets of the 3D rendering;
receive, in real-time, the at least one measured response of the user to the 3D asset,
update the user-specific personal performance profile based on the at least one measured response of the user to the 3D asset;
compare, in real-time, the received at least one measured response of the user at a given time with a corresponding predetermined user response value at a corresponding predetermined time value to the 3D asset;
generate a dynamic decision matrix customized for the user based on the user-specific personal performance profile and a neural efficiency score of the user, and
modify, in real-time, at least one of the plurality of 3D assets of the 3D rendering based on the dynamic decision matrix customized for the user and the user response comparison.

### Claim 13 (depends on 12)

The computer-implemented method of claim 12, wherein the one or more sensors comprises:
an active sensor configured to measure a physical response of the user to the 3D asset; or
a cognitive or physiological sensor configured to measure a passive cognitive or passive physiological response of the user to the 3D asset.

### Claim 14 (depends on 11)

The computer-implemented method of claim 11, wherein the one or more sensors comprises:
an active sensor configured to measure a physical response of the use r to the 3D asset; and
a cognitive or physiological sensor configured to measure a passive cognitive or passive physiological response of the user to the 3D asset, and
wherein the set of instructions stored on the computer readable-medium being further operable when executed on the processor to generate the neural efficiency score of the user based on:
a proficiency score determined, in real-time, based on a comparison between the measured physical response of the user to the 3D asset and a corresponding measurement of correctness, and the measured passive cognitive or passive physiological response of the user to the 3D asset.

### Claim 15 (depends on 12)

The computer-implemented method of claim 12, wherein the predetermined user response value is determined, in part, based on an experience level of the user, a training level of the user, or a neural efficiency score of the user.

### Claim 16 (depends on 14)

The computer-implemented method of claim 14, wherein the set of instructions stored on the computer-readable medium being further operable when executed on the processor to: modify, in real-time, the at least one of the plurality of 3D assets of the rendering based on a comparison between the generated user neural efficiency score and a neural efficiency threshold.

### Claim 17 (depends on 12)

The computer-implemented method of claim 12, wherein the generated personal performance profile specific to the user is further based on one or more of: demographic information of the user, addiction history of the user, an emotional and personality profile of the user, program objectives for the 3D rendering, and feedback from a trainer observing the user's performance.

### Claim 18 (depends on 14)

The computer-implemented method of claim 14, wherein the at least one of the plurality of 3D assets of the 3D rendering is further modified to maintain the measured passive cognitive or passive physiological response of the user at a predetermined peak performance stress level.

### Claim 19 (depends on 12)

The computer-implemented method for immersive training of claim 12, wherein the 3D rendering that is being rendered on the VR display is during a VR scenario, and wherein the set of instructions stored on the computer-readable medium is further operable to:
receive the at least one measured response of the user during the VR scenario,
compare, during the VR scenario, the received at least one measured response of the user at a given time with the corresponding predetermined user response value to the 3D asset at a corresponding predetermined time value of the VR scenario; and
modify the at least one of the plurality of 3D assets of the 3D rendering during the VR scenario.

### Claim 20

A non-transitory computer-readable medium comprising a set of instructions stored thereon, the set of instructions being operable when executed on a processor to:
receive, in real-time, from one or more sensors, at least one measured response ofa user to a three-dimensional (3D) asset ofa 3D rendering that is being rendered on a virtual reality (VR) display, the 3D rendering comprising a plurality of 3D assets;
generate a personal performance profile specific to the user and based on prior stored measured user responses to one or more of the 3D assets of the plurality of assets of the 3D rendering;
update the user-specific personal performance profile based on the at least one measured response of the user to the 3D asset;
compare, in real-time, the received at least one measured response of the user at a given time with a corresponding predetermined user response value to the 3D asset at a corresponding predetermined time value to the 3D asset;
generate a dynamic decision matrix customized for the user based on the user-specific personal performance profile and a neural efficiency score of the user; and
modify, in real-time, at least one of the plurality of 3D assets of the 3D rendering based on the dynamic decision matrix customized for the user and the user response comparison.