# US 11,308,938 B2 · Synthesizing speech recognition training data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,938 B2
- Document ID: US11308938B2
- Status: Granted
- Kind: B2
- Application number: 16/704,216
- Filed: 2019-12-05
- Published: 2021-06-10
- Granted: 2022-04-19
- Earliest claimed: 2019-12-05
- Assignees: SoundHound, Inc.
- Applicants: SoundHound, Inc.
- Inventors: Maisy Wieman; Jonah Probell; Sudharsan Krishnaswamy
- Web page: https://patentgrep.com/patent/US11308938B2

## CPC

- G10L 15/063: {Training}
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 13/02: Methods for producing synthetic speech; Speech synthesisers
- G10L 15/16: using artificial neural networks
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/187: Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
- G10L 15/197: Probabilistic grammars, e.g. word n-grams
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 13/00: Speech synthesis; Text to speech systems
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

To train a speech recognizer, such as for recognizing variables in a neural speech-to-meaning system, compute, within an embedding space, a range of vectors of features of natural speech. Generate parameter sets for speech synthesis and synthesis speech according to the parameters. Analyze the synthesized speech to compute vectors in the embedding space. Using a cost function that favors an even spread (minimal clustering) generates a multiplicity of speech synthesis parameter sets. Using the multiplicity of parameter sets, generate a multiplicity of speech of known words that can be used as training data for speech recognition.

## Claims

### Claim 1

A method of training a speech-to-meaning model, the method comprising:
determining a multiplicity of words that may be values of a variable in a phrasing of an intent;
determining a multiplicity of parameter sets representative of the voices of users of a virtual assistant, the parameter sets being chosen with an approximately even distribution within a range in an embedding space defined by vectors computed from recorded speech;
synthesizing a multiplicity of speech audio segments for the multiplicity of words, the segments being synthesized according to the multiplicity of parameter sets; and
training, using the synthesized speech audio segments, a variable recognizer that is able to compute a probability of the presence of any of the multiplicity of words in speech audio.

### Claim 2 (depends on 1)

The method of claim 1 further comprising:
training an intent recognizer on segments of speech audio of a phrasing, wherein an input to the intent recognizer is a probability output from the variable recognizer.

### Claim 3 (depends on 2)

The method of claim 2 further comprising:
determining a context of the variable within the phrasing with respect to adjacent phonetic information,
wherein the context is a further parameter of the speech synthesis.

### Claim 4 (depends on 2)

The method of claim 2 further comprising:
determining a context of the variable within the phrasing with respect to emphasis,
wherein the context is a further parameter of the speech synthesis.

### Claim 5

A method of computing a plurality of speech synthesis parameter sets representative of a diversity of voices, the method comprising:
procuring a multiplicity of speech audio recordings of natural people representative of the diversity of voices;
analyzing the recordings to compute recorded speech vectors within an embedding space of voice features;
computing a region representing a range of recorded speech vectors within the embedding space; and
learning the plurality of speech synthesis parameter sets by gradient descent according to a loss function computed by:
synthesizing speech segments according to parameter sets in the plurality of parameter sets;
analyzing the synthesized speech segments to compute synthesized speech vectors in the space; and
computing a loss in proportion to a clustering of the synthesized speech vectors within the space.

### Claim 6 (depends on 5)

The method of claim 5 wherein the embedding space is learned.

### Claim 7 (depends on 5)

The method of claim 5 further comprising:
synthesizing segments of speech of an enumerated word according to the speech synthesis parameter sets; and
training a variable recognizer, wherein the training data includes the synthesized segments of speech of the enumerated word.