# US 2020/0394935 A1 · Augmented reality techniques for simultaneously learning multiple languages

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0394935 A1
- Document ID: US20200394935A1
- Status: Published application
- Kind: A1
- Application number: 16/441,825
- Filed: 2019-06-14
- Published: 2020-12-17
- Earliest claimed: 2019-06-14
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Shubhadip Ray; Sarbajit K. Rakshit; Avik Sanyal; Craig M. Trim
- Web page: https://patentgrep.com/patent/US20200394935A1

## CPC

- G09B 19/06: Foreign languages (with audible presentation of material to be studied G09B5/04)
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G06T 19/003: {Navigation within 3D models or images}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 20/20: in augmented reality scenes
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V 40/174: {Facial expression recognition}
- G08B 7/066: {guiding along a path, e.g. evacuation path lighting strip}
- G09B 5/02: with visual presentation of the material to be studied, e.g. using film strip
- G10L 15/005: {Language recognition}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06F 40/205: Parsing
- G06F 40/263: Language identification
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A system and method for using augmented reality for assisting speech development of multiple languages includes analyzing speech recorded in an environment to detect a word used in a conversation between a first speaker and a second speaker that is in a language different from other words in the conversation, wherein a target user is in the environment with the first speaker and the second speaker, isolating an object associated with the word within an augmented reality environment of the target user located in the environment, determining a confusion level of the target user based on a use of the word in the conversation, and implementing, by the processor, an augmented reality technique based on the confusion level of the target user.

## Claims

### Claim 1

A method comprising:
analyzing, by a processor of a computing system, speech recorded in an environment to detect a word used in a conversation between a first speaker and a second speaker that is in a language different from other words in the conversation, wherein a target user is in the environment with the first speaker and the second speaker;
isolating, by the processor, an object associated with the word within an augmented reality environment of the target user located in the environment;
determining, by the processor, a confusion level of the target user based on a use of the word in the conversation; and
implementing, by the processor, an augmented reality technique based on the confusion level of the target user.

### Claim 2 (depends on 1)

The method of claim 1, wherein the augmented reality technique includes:
rendering, by the processor, a text translation of the word in the augmented reality of the target user, which assists the target user in learning the word in multiple languages.

### Claim 3 (depends on 1)

The method of claim 1, wherein the augmented reality technique includes:
initiating, by the processor, an augmented reality game within the augmented reality of the target user to distract the target user from the conversation.

### Claim 4 (depends on 3)

The method of claim 3, wherein the initiating the augmented reality game includes:
dynamically creating, by the processor, a gaming context within the augmented reality of the target user that navigates the target user away from the first speaker and the second speaker;
wherein a distance that the target user is navigated away from the conversation correlates to a loudness of the conversation.

### Claim 5 (depends on 1)

The method of claim 1, wherein the isolating includes:
searching, by the processor, for an object in the environment that correlates to the word detected as a function of the analyzing, by processing environmental data received from a plurality of input mechanisms installed in an environment; and
alerting, by the processor, the target user to a location of the object within the environment.

### Claim 6 (depends on 1)

The method of claim 1, wherein the isolating includes:
generating, by the processor, an object in the augmented reality environment of the target user that correlates to the word detected as a function of the analyzing.

### Claim 7 (depends on 1)

The method of claim 1, wherein the confusion level of the target user is predicted by detecting a mobility behavior and a facial expression of the target user in response to isolating the object associated with the word.

### Claim 8

(canceled)

### Claim 9 (depends on 8)

The computing system of claim 8, wherein the augmented reality technique includes:
rendering, by the processor, a text translation of the word in the augmented reality of the target user, which assists the target user in learning the word in multiple languages.

### Claim 10 (depends on 8)

The computing system of claim 8, wherein the augmented reality technique includes:
initiating, by the processor, an augmented reality game within the augmented reality of the target user to distract the target user from the conversation.

### Claim 11 (depends on 10)

The computing system of claim 10, wherein the initiating the augmented reality game includes:
dynamically creating, by the processor, a gaming context within the augmented reality of the target user that navigates the target user away from the first speaker and the second speaker;
wherein a distance that the target user is navigated away from the conversation correlates to a loudness of the conversation.

### Claim 12 (depends on 8)

The computing system of claim 8, wherein the isolating includes:
searching, by the processor, for an object in the environment that correlates to the word detected as a function of the analyzing, by processing environmental data received from a plurality of input mechanisms installed in an environment; and
alerting, by the processor, the target user to a location of the object within the environment.

### Claim 13 (depends on 8)

The computing system of claim 8, wherein the isolating includes:
generating, by the processor, an object in the augmented reality environment of the target user that correlates to the word detected as a function of the analyzing.

### Claim 14 (depends on 8)

The computing system of claim 8, wherein the confusion level of the target user is predicted by detecting a mobility behavior and a facial expression of the target user in response to isolating the object associated with the word.

### Claim 15 (depends on 8)

The computing system of claim 8, further comprising: creating, by the processor, a profile of the target user over time, and comparing the profile of the target user to a database of profiles of other target users to assist in determining the confusion level of the target user.

### Claim 16

A computer program product, comprising a computer readable hardware storage device storing a computer readable program code, the computer readable program code comprising an algorithm that when executed by a computer processor of a computing system implements a method comprising:
analyzing, by a processor of a computing system, speech recorded in an environment to detect a word used in a conversation between a first speaker and a second speaker that is in a language different from other words in the conversation, wherein a target user is in the environment with the first speaker and the second speaker;
isolating, by the processor, an object associated with the word within an augmented reality environment of the target user located in the environment;
determining, by the processor, a confusion level of the target user based on a use of the word in the conversation; and
implementing, by the processor, an augmented reality technique based on the confusion level of the target user.

### Claim 17 (depends on 16)

The computer program product of claim 16, wherein the augmented reality technique includes:
rendering, by the processor, a text translation of the word in the augmented reality of the target user, which assists the target user in learning the word in multiple languages.

### Claim 18 (depends on 16)

The computer program product of claim 16, wherein the augmented reality technique includes:
initiating, by the processor, an augmented reality game within the augmented reality of the target user to distract the target user from the conversation.

### Claim 19 (depends on 18)

The computer program product of claim 18, wherein the initiating the augmented reality game includes:
dynamically creating, by the processor, a gaming context within the augmented reality of the target user that navigates the target user away from the first speaker and the second speaker;
wherein a distance that the target user is navigated away from the conversation correlates to a loudness of the conversation.

### Claim 20 (depends on 16)

The computer program product of claim 16, wherein the isolating includes:
generating, by the processor, an object in the augmented reality environment of the target user that correlates to the word detected as a function of the analyzing.
