# US 2020/0286475 A1 · Two-person Automatic Speech Recognition Training To Interpret Unknown Voice Inputs

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0286475 A1
- Document ID: US20200286475A1
- Status: Published application
- Kind: A1
- Application number: 16/297,298
- Filed: 2019-03-08
- Published: 2020-09-10
- Earliest claimed: 2019-03-08
- Assignees: Motorola Mobility LLC
- Applicants: Motorola Mobility LLC
- Inventors: Zhengping Ji; Rachid M. Alameh
- Web page: https://patentgrep.com/patent/US20200286475A1

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06N 3/09: Supervised learning
- G06N 5/02: Knowledge representation; Symbolic representation
- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G06N 20/00: Machine learning
- G10L 15/063: {Training}
- G10L 15/07: to the speaker
- G10L 15/18: using natural language modelling
- G10L 25/66: for extracting parameters related to health condition (detecting or measuring for diagnostic purposes A61B5/00)
- G06N 3/08: Learning methods
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06N 20/20: Ensemble learning
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A system receives a first voice input from a first user, such as a baby or a person who has had a stroke. Although the first user intends to communicate a particular meaning, the first voice input is not in a language that is known to the system and thus the system does not know the particular meaning that the first user intended. After receiving the first voice input, a second voice input is received from a second user. This second voice input is in a language that is known to the system. The system determines a meaning of the second voice input, associates this meaning with the first voice input, and uses this association to train a machine learning system. This machine learning system is used to attempt to understand the meaning of subsequent voice inputs received from the first user.

## Claims

### Claim 1

A method implemented in an unknown voice input interpretation system of a computing device, the method comprising:
receiving a first voice input from a first user;
determining that the first voice input is not in a known language to the unknown voice input interpretation system;
receiving, subsequent to receiving the first voice input, a second voice input from a second user;
determining a meaning of the second voice input;
associating the meaning of the second voice input with the first voice input;
using the first voice input and the meaning of the second voice input to train a machine learning system of the unknown voice input interpretation system to understand voice inputs from the first user;
receiving a third voice input from the first user; and
using the machine learning system to determine a meaning of the third voice input.

### Claim 2 (depends on 1)

The method as recited in claim 1, the method further comprising taking an action in response to the third voice input based on the meaning of the third voice input.

### Claim 3 (depends on 1)

The method as recited in claim 1, wherein the associating comprises associating the meaning of the second voice input with the first voice input only in response to the second voice input being received within a threshold amount of time of receiving the first voice input.

### Claim 4 (depends on 1)

The method as recited in claim 1, wherein the associating comprises associating the meaning of the second voice input with the first voice input only in response to determining that there is a known relationship between the first user and the second user.

### Claim 5 (depends on 4)

The method as recited in claim 4, further comprising determining whether there is a known relationship between the first user and the second user based on a voice ID of the first user, a voice ID of the second user, and a knowledge graph.

### Claim 6 (depends on 1)

The method as recited in claim 1, wherein the associating comprises associating the meaning of the second voice input with the first voice input only in response to determining that the first user and the second user are in close proximity to one another.

### Claim 7 (depends on 1)

The method as recited in claim 1, wherein the determining the meaning of the second voice input comprises using sentiment analysis to determine whether the second voice input is in response to the first user, and wherein the associating comprises associating the meaning of the second voice input with the first voice input only in response to determining that the second voice input is in response to the first user.

### Claim 8 (depends on 1)

The method as recited in claim 1, the method further comprising:
receiving a fourth voice input from the first user that is not in a known language to the unknown voice input interpretation system;
storing the fourth voice input;
subsequently receiving a fifth voice input from the first user that is not in a known language to the unknown voice input interpretation system;
determining a meaning of the fifth voice input;
determining that the fourth voice input is close to the fifth voice input;
using the fourth voice input and the meaning of the fifth voice input to train the machine learning system to understand voice inputs from the first user; and
using the fifth voice input and the meaning of the fifth voice input to train the machine learning system to understand voice inputs from the first user.

### Claim 9

A computing device comprising:
a processor; and
a computer-readable storage medium having stored thereon multiple instructions that implement an unknown voice input interpretation system and that, responsive to execution by the processor, cause the processor to perform acts comprising:
determining that a first voice input received from a first user is not in a known language to the unknown voice input interpretation system;
determining a meaning of a second voice input received, subsequent to receipt of the first voice input, from a second user;
associating the meaning of the second voice input with the first voice input;
using the first voice input and the meaning of the second voice input to train a machine learning system of the unknown voice input interpretation system to understand voice inputs from the first user; and
using the machine learning system to determine a meaning of a third voice input received from the first user.

### Claim 10 (depends on 9)

The computing device as recited in claim 9, the acts further comprising taking an action in response to the third voice input based on the meaning of the third voice input.

### Claim 11 (depends on 9)

The computing device as recited in claim 9, wherein the associating comprises associating the meaning of the second voice input with the first voice input only in response to the second voice input being received within a threshold amount of time of receiving the first voice input.

### Claim 12 (depends on 9)

The computing device as recited in claim 9, wherein the associating comprises associating the meaning of the second voice input with the first voice input only in response to determining that there is a known relationship between the first user and the second user.

### Claim 13 (depends on 12)

The computing device as recited in claim 12, further comprising determining whether there is a known relationship between the first user and the second user based on a voice ID of the first user, a voice ID of the second user, and a knowledge graph.

### Claim 14 (depends on 9)

The computing device as recited in claim 9, wherein the associating comprises associating the meaning of the second voice input with the first voice input only in response to determining that the first user and the second user are in close proximity to one another.

### Claim 15 (depends on 9)

The computing device as recited in claim 9, wherein the determining the meaning of the second voice input comprises using sentiment analysis to determine whether the second voice input is in response to the first user, and wherein the associating comprises associating the meaning of the second voice input with the first voice input only in response to determining that the second voice input is in response to the first user.

### Claim 16 (depends on 9)

The computing device as recited in claim 9, the acts further comprising:
receiving a fourth voice input from the first user that is not in a known language to the unknown voice input interpretation system;
storing the fourth voice input;
subsequently receiving a fifth voice input from the first user that is not in a known language to the unknown voice input interpretation system;
determining a meaning of the fifth voice input;
determining that the fourth voice input is close to the fifth voice input;
using the fourth voice input and the meaning of the fifth voice input to train the machine learning system to understand voice inputs from the first user; and
using the fifth voice input and the meaning of the fifth voice input to train the machine learning system to understand voice inputs from the first user.

### Claim 17

An unknown voice input interpretation system comprising:
a speech recognition module, implemented at least in part in hardware, configured to determine that a first voice input received from a first user is not in a known language to the unknown voice input interpretation system;
a natural language understanding module, implemented at least in part in hardware, configured to determine a meaning of a second voice input received, subsequent to receipt of the first voice input, from a second user;
a training module, implemented at least in part in hardware, configured to associate the meaning of the second voice input with the first voice input, and to use the first voice input and the meaning of the second voice input to train a machine learning system of the unknown voice input interpretation system to understand voice inputs from the first user; and
the machine learning system, implemented at least in part in hardware, configured to determine a meaning of a third voice input received from the first user.

### Claim 18 (depends on 17)

The system as recited in claim 17, wherein the training module is further configured to associate the meaning of the second voice input with the first voice input only in response to the second voice input being received within a threshold amount of time of receiving the first voice input.

### Claim 19 (depends on 17)

The system as recited in claim 17, wherein the training module is further configured to associate the meaning of the second voice input with the first voice input only in response to determining that there is a known relationship between the first user and the second user.

### Claim 20 (depends on 17)

The system as recited in claim 17, wherein the natural language understanding module is further configured to use sentiment analysis to determine whether the second voice input is in response to the first user, and wherein the training module is further configured to associate the meaning of the second voice input with the first voice input only in response to the natural language understanding module determining that the second voice input is in response to the first user.
