# US 2017/0287465 A1 · Speech Recognition and Text-to-Speech Learning System

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0287465 A1
- Document ID: US20170287465A1
- Status: Published application
- Kind: A1
- Application number: 15/087,696
- Filed: 2016-03-31
- Published: 2017-10-05
- Earliest claimed: 2016-03-31
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Pei Zhao; Kaisheng Yao; Max Leung; Bo Yan
- Web page: https://patentgrep.com/patent/US20170287465A1

## CPC

- G10L 13/10: Prosody rules derived from text; Stress or intonation
- G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- G10L 13/086: {Detection of language}
- G10L 15/063: {Training}
- G10L 15/07: to the speaker
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

An example text-to-speech learning system performs a method for generating a pronunciation sequence conversion model. The method includes generating a first pronunciation sequence from a speech input of a training pair and generating a second pronunciation sequence from a text input of the training pair. The method also includes determining a pronunciation sequence difference between the first pronunciation sequence and the second pronunciation sequence; and generating a pronunciation sequence conversion model based on the pronunciation sequence difference. An example speech recognition learning system performs a method for generating a pronunciation sequence conversion model. The method includes extracting an audio signal vector from a speech input and applying an audio signal conversion model to the audio signal vector to generate a converted audio signal vector. The method also includes adapting an acoustic model based on the converted audio signal vector to generate an adapted acoustic model.

## Claims

### Claim 1

A text-to-speech learning system, the system comprising:
at least one processor; and
memory, operatively connected to the at least one processor and storing instructions that, when executed by the at least processor, cause the at least one processor to perform a method for generating a pronunciation sequence conversion model, the method comprising:
generating a first pronunciation sequence from a speech input of a training pair;
generating a second pronunciation sequence from a text input of the training pair;
determining a pronunciation sequence difference between the first pronunciation sequence and the second pronunciation sequence; and
generating a pronunciation sequence conversion model based on the pronunciation sequence difference.

### Claim 2 (depends on 1)

The text-to-speech learning system of claim 1, wherein the method further comprises extracting an audio signal vector from the speech input of the training pair, and wherein the first pronunciation sequence is generated based on the extracted audio signal vector.

### Claim 3 (depends on 1)

The text-to-speech learning system of claim 1, wherein the pronunciation sequence conversion model comprises a recursive neural network.

### Claim 4 (depends on 1)

The text-to-speech learning system of claim 1, wherein determining a pronunciation sequence difference between the first pronunciation sequence and the second pronunciation sequence comprises aligning the first pronunciation sequence with the second pronunciation sequence.

### Claim 5 (depends on 1)

The text-to-speech learning system of claim 1, wherein the first pronunciation sequence comprises a sequence of pronunciation signals.

### Claim 6 (depends on 1)

The text-to-speech learning system of claim 1, wherein the pronunciation sequence conversion model is generated based on a plurality of pronunciation sequences from a plurality of training pairs, the plurality of pronunciation sequence differences including the pronunciation sequence difference.

### Claim 7 (depends on 6)

The text-to-speech learning system of claim 6, wherein the plurality of pronunciation sequence differences comprises at least one pronunciation sequence difference generated from a training pair selected from a text-to-speech training corpus and at least one pronunciation sequence difference generated from a training pair selected from a speech-recognition training corpus.

### Claim 8 (depends on 1)

The text-to-speech learning system of claim 1, wherein the pronunciation sequence generator model is configured to be used by a text-to-speech system to synthesize speech.

### Claim 9

A speech recognition learning system, the system comprising:
at least one processor; and
memory, operatively connected to the at least one processor and storing instructions that, when executed by the at least processor, cause the at least one processor to perform a method for generating a pronunciation sequence conversion model, the method comprising:
extracting an audio signal vector from a speech input;
applying an audio signal conversion model to the audio signal vector to generate a converted audio signal vector; and
adapting an acoustic model based on the converted audio signal vector to generate an adapted acoustic model.

### Claim 10 (depends on 9)

The speech recognition learning system of claim 9, wherein the adapted acoustic model is configured to be used by a speech-recognition system to recognize speech from a user.

### Claim 11 (depends on 9)

The speech recognition learning system of claim 9, wherein the method further comprises generating an audio vector conversion model based on a plurality of training pairs.

### Claim 12 (depends on 11)

The speech recognition learning system of claim 11, wherein the method further comprises comparing an audio signal vector extracted from a speech input of a training pair of the plurality of training pairs to a second audio signal vector generated from the text input of the training pair.

### Claim 13 (depends on 12)

The speech recognition learning system of claim 12, wherein the method further comprises determining a difference between the extracted audio signal vector and the second audio signal vector.

### Claim 14 (depends on 13)

The speech recognition learning system of claim 13, wherein determining the difference between the extracted audio signal vector and the second audio signal vector comprises aligning the extracted audio signal vector with the second audio signal vector.

### Claim 15 (depends on 12)

The speech recognition learning system of claim 12, wherein the second audio signal vector is generated by extracting an audio signal vector from synthesized speech based on the text input of the training pair.

### Claim 16 (depends on 11)

The speech recognition learning system of claim 11, wherein the audio signal vector conversion model is configured to be used by a speech recognition system to recognize speech from a user.

### Claim 17 (depends on 9)

The speech recognition learning system of claim 9, wherein the adapted acoustic model is generated by adapting a plurality of extracted audio signal vectors from a plurality of speech inputs, the plurality of extracted audio signal vectors including the audio signal vector.

### Claim 18 (depends on 17)

The speech recognition learning system of claim 17, wherein the plurality of speech inputs comprise at least one speech input from a training pair in a text-to-speech training corpus and at least one speech input from a training pair in a speech-recognition training corpus.

### Claim 19

A method for generating a text-to-speech model and a speech-recognition model, the method comprising:
generating a pronunciation sequence conversion model based on a plurality of pronunciation sequence differences, wherein each of the pronunciation sequence differences is associated with a training pair from a plurality of training pairs and each of the pronunciation sequence differences is generated by comparing a first pronunciation sequence generated from a speech input of a training pair associated with the pronunciation sequence difference to a second pronunciation sequence generated from the text input of the training pair associated with the pronunciation sequence difference; and
adapting an acoustic model based on a plurality of converted audio signal vectors, wherein each of the converted audio signal vectors is associated with a speech input from a plurality of speech inputs and the each of the converted audio signal vectors is generated by extracting an audio signal vector from the speech input associated with the converted audio signal vector and applying an audio signal vector conversion model to the extracted audio signal vector.

### Claim 20 (depends on 19)

The method of claim 19 further comprising:
generating an audio vector conversion model based on a plurality of audio signal vector differences, wherein each of the audio signal vectors is associated with a training pair from the plurality of training pairs and each of the audio signal vector differences is generated by comparing an audio signal vector extracted from the speech input of the training pair associated with the audio signal vector difference and a second audio signal vector generated from the text input of the training pair associated with the audio signal vector difference.
