# US 9,390,630 B2 · Accelerated learning, entertainment and cognitive therapy using augmented reality comprising combined haptic, auditory, and visual stimulation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,390,630 B2
- Document ID: US9390630B2
- Status: Granted
- Kind: B2
- Application number: 14/269,133
- Filed: 2014-05-03
- Published: 2015-11-05
- Granted: 2016-07-12
- Earliest claimed: 2013-05-03
- Applicants: John James Daniels
- Inventors: John James Daniels
- Web page: https://patentgrep.com/patent/US9390630B2

## CPC

- G09B 15/00: Teaching music (metronomes G04F5/02)
- A61M 21/00: Other devices or methods to cause a change in the state of consciousness; Devices for producing or ending sleep by mechanical, optical, or acoustical means, e.g. for hypnosis
- G09B 5/065: {Combinations of audio and video presentations, e.g. videotapes, videodiscs, television systems}
- G09B 9/00: Simulators for teaching or training purposes (for the use of weapons F41; computing aspects G06; {protocols for games, networked simulations or virtual reality H04L67/131})
- G09B 9/003: {for military purposes and tactics}
- G09B 9/06: for teaching control of ships, boats, or other waterborne vehicles
- G09B 9/063: {by using visual displays}
- G09B 9/08: for teaching control of aircraft, e.g. Link trainer
- G09B 9/52: for teaching control of an outer space vehicle
- G09B 15/003: {with indication of the keys or strings to be played on instruments}
- G09B 15/06: Devices for exercising or strengthening fingers or arms; Devices for holding fingers or arms in a proper position for playing (for teaching typing G09B13/00; exercising apparatus for developing or strengthening the muscles for physical training A63B21/00, A63B23/00)
- G09B 19/00: Teaching not covered by other main groups of this subclass (teaching or practice apparatus for gun-aiming or gun-laying F41G3/26)
- G09B 19/003: {Repetitive work cycles; Sequence of movements}
- G09B 19/0038: {Sports}
- G09B 19/22: Games, e.g. card games
- G09B 21/00: Teaching, or communicating with, the blind, deaf or mute (audible presentation of material to be studied G09B5/04; devices or methods for replacing direct visual or auditory perception by another kind of perception A61F9/08, A61F11/04; audible indication of meter readings or of colour G01D7/12; watches for blind persons G04B25/02; speech analysis, speech recognition G10L; sound-recording or reproducing, per se G11B)
- G10H 1/18: Selecting circuits
- G16H 20/30: relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
- A61M 21/0094: {Isolation chambers used therewith, i.e. for isolating individuals from external stimuli (other treatment rooms or enclosures A61G10/00)}
- A61M 2021/0016: {by the smell sense}
- A61M 2021/0022: {by the tactile sense, e.g. vibrations}
- A61M 2021/0027: {by the hearing sense}
- A61M 2021/005: {images, e.g. video}
- A61M 2021/0072: {with application of electrical currents}
- A61M 2021/0077: {with application of chemical or pharmacological stimulus}
- A61M 2205/507: Head Mounted Displays [HMD]
- A61M 2210/0612: Eyes
- A61M 2210/0662: Ears
- A61M 2210/083: Arms
- A61M 2210/086: Legs
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- A61M: DEVICES FOR INTRODUCING MEDIA INTO, OR ONTO, THE BODY (introducing media into or onto the bodies of animals A61D7/00; means for inserting tampons A61F13/26; devices for administering food or medicines orally A61J; containers for collecting, storing or administering blood or medical fluids A61J1/05); DEVICES FOR TRANSDUCING BODY MEDIA OR FOR TAKING MEDIA FROM THE BODY (surgery A61B; chemical aspects of surgical articles A61L); DEVICES FOR PRODUCING OR ENDING SLEEP OR STUPOR {(Electrotherapy, e.g. producing anaesthesia by the use of alternating or intermittent currents A61N1/36021)}
- G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

An accelerated learning and rehabilitation system for teaching the performance of a musical instrument, a remotely operated system, a sport, a weapon, and for brain rehabilitation and other uses includes generating sensory cues including auditory, haptic and visual sensory cues capable of being perceived by a user. The generated sensory cues are applied to the user and are dependent on a position of at least one body member of a performer relative to a performance element of a performance object with which an event is performed. The sensory cues are effective for stimulating a various processing center of a brain of the user so that user learns how to position his body member corresponding to the position of the performer of the event. The sensory cues can include visual sensory cues effective for stimulating the visual processing center of the brain of the user. The visual sensory cues are synchronized with the other applied sensory cues so that the position of the body member of the performer is virtually visually indicated in synchronization with the other sensory cues so that the visual processing center is stimulated with a visual sensory cue in synchronization with the stimulation of other processing centers corresponding to the other sensory cues for teaching the user to perform a version of the event.

## Claims

### Claim 1

An apparatus, comprising:
at least one processors; and
at least one memory including computer program code,
the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following;
detect a change in position of a body member of a performer relative to a performance element of a performance object with which an event is to be performed;
generate a signal dependent on the detected change in position of the body member;
record the signal so that a first sensory cue can be determined to indicate the change in position of the body member relative to the element of the performance object during a learning session;
determine the first sensory cue dependent on the recorded signal; and
apply the first sensory cue to indicate to a user during the learning session the change in position of the body member of the performer during the performance of the event, the first sensory cue being effective for stimulating a first processing center of a brain of the user,
wherein the first sensory cue is an applied electrical signal dependent on the recorded signal effective to stimulate at least one of muscles and nerves of the user during the learning session to indicate to the user the position of the body member of the performer during the performance.

### Claim 2 (depends on 1)

An apparatus according to claim 1; wherein the body member is at least one of a hand and at least one finger of the performer; and wherein the event is piece of music to be performed on a musical instrument, the performance object is the musical instrument and the performance element is at least one of a key, a string and a valve of the musical instrument.

### Claim 3 (depends on 1)

An apparatus according to claim 1: wherein the change in position of the body member is detected from electrical impulses received from the body member.

### Claim 4 (depends on 1)

An apparatus according to claim 1:
wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to further:
record, during the performance of the event, at least one of haptic, auditory and visual data related to the performance of the event;
determine at least a second sensory cue dependent on the recorded at least one of haptic, auditory and visual data; and
apply, during the learning session, the at least said second sensory cue synchronized with the application of the first sensory cue effective for stimulating at least a second processing center of the brain of the user.

### Claim 5 (depends on 1)

An apparatus according to claim 1: wherein the applied electrical signal causes a sensation or contraction in the muscles of the user.

### Claim 6 (depends on 1)

An apparatus according to claim 1: wherein the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to further perform; detect electrical impulses of the user during the learning session.

### Claim 7

A method, comprising:
detecting a change in position of a body member of a performer relative to a performance element of a performance object with which an event is to be performed;
generating a signal dependent on the detected change in position of the body member; and
recording the signal so that a first sensory cue can be determined to indicate the change in position of the body member relative to the performance element of the performance object during a learning session;
determining the first sensory cue dependent on the recorded signal; and applying the first sensory cue to indicate to a user learning session the learning session the change in position of the body member of the performer during the performance of the event,the first sensory cue being effective for stimulating a first processing center of the user; wherein the
first sensory cue is one of a haptic, auditory and visual sensory cue effective for stimulating a first processing center of a brain of the user; and
generating a visual sensory cue capable of being displayed to the user on a video display device, the visual sensory cue providing a virtual visual indication to the user of the change in position of the body member of the performer during the performance, the visual sensory cue being effective for stimulating the visual processing center of the brain of the user, the visual sensory cue being synchronized with the first sensory cue so that the change in position of body member of the performer is virtually visually indicated in synchronization with the first sensory cue and so that the visual processing center is stimulated with a visual sensory cue in synchronization, stimulating the first processing center,
wherein thesnthat5iDimuiatio of the first processing center and the visual processing center is effective for teaching the user to perform a version of the event.

### Claim 8 (depends on 7)

A method according to claim 7: wherein the body member is at least one of a hand and at least one finger of the performer; and wherein the event is piece of music to be performed on a musical instrument, the performance object is the musical instrument and the performance element is at least one of a key, a string and a valve of the musical instrument.

### Claim 9 (depends on 7)

A method according to claim 7: wherein the first sensory cue is one of a haptic, auditory and visual sensory cue; and further comprising:
recording, during the performance of the event, at least one of haptic, auditory and visual data related to the performance of the event;
determining at least a second sensory cue dependent on the recorded at least one of haptic, auditory and visual data; and
applying, during the learning session, the at least said second sensory cue synchronized with the application of the that sensory cue effective for stimulating at least a second processing center of the brain of the user.

### Claim 10 (depends on 7)

A method according to claim 7: wherein the applied electrical signal causes a sensation or contraction in the muscles of the user.

### Claim 11 (depends on 7)

A method according to claim 7: wherein the applied first sensory cue stimulates at least one of nerves and muscles of the user.

### Claim 12 (depends on 7)

A method according to claim 7: wherein the applied first sensory cue comprises at least one of a vibration and an electrical signal.

### Claim 13 (depends on 7)

A method according to claim 7: wherein the applied first sensory cue comprises an audio sensory cue.

### Claim 14 (depends on 7)

A method according to claim 7: further comprising detecting electrical impulses of the user during the learning session.

### Claim 15

An apparatus, comprising:
at least one processors; and
at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following:
receive a recorded signal, the recorded signal being a signal generated for recording during a performance of an event dependent on a detected change in position of a body member of a performer relative to a performance element of a performance object with which the event was performed;
determine the a first sensory cue dependent on the recorded signal;
apply the first sensory cue to indicate to a user during a learning session the change in position of the body member of the performer during the performance of the event, wherein the first sensory cue is one of a haptic, auditory and visual sensory cue effective for stimulating a first processing center of a brain of the user;
receive a visual sensory data;
determine from the received visual sensory data a visual sensory cue capable of being displayed to the user on, a video display device; and
display the visual sensory cue to the user during the learning session wherein the visual sensory cue provides a virtual visual indication to the user of the change in position of the body member of the performer during the performance, the visual sensory cue being effective for stimulating the visual processing center of the brain of the user, the visual sensory cue being synchronized with the first sensory cue so that the chance in position of body member of the performer is virtually indicated in synhronization with the first sensory cue and so that the visual processing center is stimulated with a visual sensor cue in synchronization with a first sensory cue stimulating the first processing center, wherein the synchronization stimulation of the first processing center and the visual processing center is effective for teaching the user to perform a version of the event.

### Claim 16 (depends on 15)

An apparatus according to claim 15: wherein the change in position of the body member is detected from electrical impulses received from the body member.

### Claim 17 (depends on 15)

An apparatus according to claim 15:
wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to further:
receive at least one of haptic, auditory and visual data related to the performance of the event;
determine at least a second sensory cue dependent on the at least one of haptic, auditory and visual data; and
apply, during the learning session, the at least said second sensory cue synchronized with the application of the first sensory cue effective for stimulating at least a second processing center of the brain of the user.

### Claim 18 (depends on 17)

A method according to claim 17: wherein the applied first sensory cue stimulates at least one of nerves and muscles of the user, and wherein the applied second sensory cue comprises an audio sensory cue.

### Claim 19 (depends on 15)

An apparatus according to claim 15: wherein the body member is at least one finger of the performer, the event is piece of music to be performed on a musical instrument, the performance object is the musical instrument and the performance element is at least one of a key, a string and a valve of the musical instrument; and wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to further: provide auditory information to the user in the form of at least one note corresponding to a musical tone played on the musical instrument, said musical tone being a member of a plurality of musical tones played in series and making up the piece of music performed by the performer;
provide the first sensory cue as haptic information corresponding to the change in position of the at least one finger of the performer, the haptic information being applied to a corresponding at least one finger of the user, the haptic information being synchronized to the auditory information so that said at least one finger of the user is stimulated at the same time the musical tone is heard by the user; and
provide visual information corresponding to said at least one note, the visual information being synchronized to the auditory information so that the user sees the visual information at the same time said at least one finger is stimulated when said at least one note is heard by the user so that the user is contemporaneously stimulated with an auditory cue, a haptic cue and a visual cue indicating the at least one musical note to be played on a musical instrument when learning the plurality of musical tones played in series and making up the piece of music played by the performer on the musical instrument.

### Claim 20 (depends on 15)

An apparatus according to claim 15: wherein the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to further perform; detecting electrical impulses of the use.