# US 12,190,862 B2 · Using non-parallel voice conversion for speech conversion models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,190,862 B2 - Document ID: US12190862B2 - Status: Granted - Kind: B2 - Application number: 17/660,487 - Filed: 2022-04-25 - Published: 2023-09-21 - Granted: 2025-01-07 - Earliest claimed: 2022-03-21 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Andrew M. Rosenberg; Gary Wang; Bhuvana Ramabhadran; Fadi Biadsy - Web page: https://patentgrep.com/patent/US12190862B2 ## CPC - G10L 15/063: {Training} - G10L 13/02: Methods for producing synthetic speech; Speech synthesisers - G10L 15/197: Probabilistic grammars, e.g. word n-grams - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 19/038: Vector quantisation, e.g. TwinVQ audio - G10L 21/003: Changing voice quality, e.g. pitch or formants - G10L 15/16: using artificial neural networks - G10L 2015/0635: {updating or merging of old and new templates; Mean values; Weighting} - G10L 2019/0001: {Codebooks} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A method includes receiving a set of training utterances each including a non-synthetic speech representation of a corresponding utterance, and for each training utterance, generating a corresponding synthetic speech representation by using a voice conversion model. The non-synthetic speech representation and the synthetic speech representation form a corresponding training utterance pair. At each of a plurality of output steps for each training utterance pair, the method also includes generating, for output by a speech recognition model, a first probability distribution over possible non-synthetic speech recognition hypotheses for the non-synthetic speech representation and a second probability distribution over possible synthetic speech recognition hypotheses for the synthetic speech representation. The method also includes determining a consistent loss term for the corresponding training utterance pair based on the first and second probability distributions and updating parameters of the speech recognition model based on the consistent loss term. ## Claims ### Claim 1 A computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations comprising: receiving a set of training utterances each comprising a non-synthetic speech representation of a corresponding utterance; for each training utterance, generating a corresponding voice conversion synthetic speech representation by using a voice conversion model to convert the non-synthetic speech representation into the corresponding voice conversion synthetic representation of the corresponding utterance, the voice conversion model comprising a content encoder configured to encode each training utterance, the content encoder comprising a pretrained automatic speech recognition (ASR) encoder, wherein the non-synthetic speech representation and the voice conversion synthetic speech representation form a corresponding training utterance pair; at each of a plurality of output steps for each training utterance pair in the set of training utterance pairs: generating, for output by a speech recognition model, a first probability distribution over possible non-synthetic speech recognition hypotheses for the corresponding non-synthetic speech representation of the corresponding utterance; generating, for output by the speech recognition model, a second probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation of the corresponding utterance; and determining a consistent loss term for the corresponding training utterance pair based on the first probability distribution over possible non-synthetic speech recognition hypotheses and the second probability distribution over possible non-synthetic speech recognition hypotheses; and updating parameters of the speech recognition model based on the consistent loss term determined at each of the plurality of output steps for each training utterance pair in the set of training utterance pairs, wherein the voice conversion model comprises: the content encoder further configured to: receive, as input, input speech; and generate, as output, an initial latent representation; a vector quantization-variational autoencoder (VQ-VAE) layer configured to: receive, as input, the initial latent representation; and generate, as output, a latent representation of linguistic content extracted from the input speech that discards speaking style variations in the input speech; and a decoder conditioned on a speaker embedding and configured to generate output speech based on the latent representation of linguistic content. ### Claim 2 (depends on 1) The method of claim 1, wherein the operations further comprise, at each of the plurality of output steps for each training utterance pair in the set of training utterance pairs: generating a non-synthetic speech loss term based on the first probability distribution over possible non-synthetic speech recognition hypotheses for the corresponding non-synthetic speech representation of the corresponding utterance and a ground-truth transcription of the corresponding utterance; and generating a synthetic speech loss term based on the second probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation of the corresponding utterance and the ground-truth transcription of the corresponding utterance. ### Claim 3 (depends on 2) The method of claim 2, wherein the operations further comprise updating parameters of the speech recognition model based on the non-synthetic and synthetic speech loss terms generated at each of the plurality of output steps for each training utterance pair in the set of training utterance pairs. ### Claim 4 (depends on 1) The method of claim 1, wherein: the non-synthetic speech representation of the corresponding utterance comprises speech spoken by a source speaker and conveys an accent/locale associated with the source speaker; and the voice conversion synthetic representation of the corresponding utterance comprises synthesized speech representing the corresponding utterance that conveys an accent/local associated with a target speaker. ### Claim 5 (depends on 4) The method of claim 4, wherein the accent/local associated with the source speaker is different than the accent/local associated with the target speaker. ### Claim 6 (depends on 4) The method of claim 4, wherein the voice conversion synthetic speech representation conveys the same linguistic content as the non-synthetic speech representation of the corresponding utterance. ### Claim 7 (depends on 1) The method of claim 1, wherein the voice conversion model comprises a non-parallel voice conversion model. ### Claim 8 (depends on 1) The method of claim 1, wherein the pretrained ASR encoder was previously trained on an ASR loss for a speech recognition task. ### Claim 9 (depends on 8) The method of claim 8, wherein the parameters of the pretrained ASR encoder remain fixed while training the VQ-VAE layer and the decoder of the voice conversion model. ### Claim 10 (depends on 1) The method of claim 1, wherein the VQ-VAE layer is trained using a VQ loss based on the latent representation of linguistic content generated for each timestep, the VQ loss encouraging the VQ-VAE layer to minimize a distance between an output and a nearest codebook. ### Claim 11 (depends on 1) The method of claim 1, wherein the decoder is configured to: receive, as input, the latent representation of linguistic content for the input speech and the speaker embedding; and generate, as output, the output speech comprising a reconstruction of the input speech. ### Claim 12 A system comprising: data processing hardware; and memory hardware in communication with the data processing hardware and storing instructions that when executed by the data processing hardware causes the data processing hardware to perform operations comprising: receiving a set of training utterances each comprising a non-synthetic speech representation of a corresponding utterance; for each training utterance, generating a corresponding voice conversion synthetic speech representation by using a voice conversion model to convert the non-synthetic speech representation into the corresponding voice conversion synthetic representation of the corresponding utterance, the voice conversion model comprising a content encoder configured to encode each training utterance, the content encoder comprising a pretrained automatic speech recognition (ASR) encoder, wherein the non-synthetic speech representation and the voice conversion synthetic speech representation form a corresponding training utterance pair; at each of a plurality of output steps for each training utterance pair in the set of training utterance pairs: generating, for output by a speech recognition model, a first probability distribution over possible non-synthetic speech recognition hypotheses for the corresponding non-synthetic speech representation of the corresponding utterance; generating, for output by the speech recognition model, a second probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation of the corresponding utterance; and determining a consistent loss term for the corresponding training utterance pair based on the first probability distribution over possible non-synthetic speech recognition hypotheses and the second probability distribution over possible non-synthetic speech recognition hypotheses; and updating parameters of the speech recognition model based on the consistent loss term determined at each of the plurality of output steps for each training utterance pair in the set of training utterance pairs, wherein the voice conversion model comprises: the content encoder further configured to: receive, as input, input speech; and generate, as output, an initial latent representation; a vector quantization-variational autoencoder (VQ-VAE) layer configured to: receive, as input, the initial latent representation; and generate, as output, a latent representation of linguistic content extracted from the input speech that discards speaking style variations in the input speech; and a decoder conditioned on a speaker embedding and configured to generate output speech based on the latent representation of linguistic content. ### Claim 13 (depends on 12) The system of claim 12, wherein the operations further comprise, at each of the plurality of output steps for each training utterance pair in the set of training utterance pairs: generating a non-synthetic speech loss term based on the first probability distribution over possible non-synthetic speech recognition hypotheses for the corresponding non-synthetic speech representation of the corresponding utterance and a ground-truth transcription of the corresponding utterance; and generating a synthetic speech loss term based on the second probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation of the corresponding utterance and the ground-truth transcription of the corresponding utterance. ### Claim 14 (depends on 13) The system of claim 13, wherein the operations further comprise updating parameters of the speech recognition model based on the non-synthetic and synthetic speech loss terms generated at each of the plurality of output steps for each training utterance pair in the set of training utterance pairs. ### Claim 15 (depends on 12) The system of claim 12, wherein: the non-synthetic speech representation of the corresponding utterance comprises speech spoken by a source speaker and conveys an accent/locale associated with the source speaker; and the voice conversion synthetic representation of the corresponding utterance comprises synthesized speech representing the corresponding utterance that conveys an accent/local associated with a target speaker. ### Claim 16 (depends on 15) The system of claim 15, wherein the accent/local associated with the source speaker is different than the accent/local associated with the target speaker. ### Claim 17 (depends on 15) The system of claim 15, wherein the voice conversion synthetic speech representation conveys the same linguistic content as the non-synthetic speech representation of the corresponding utterance. ### Claim 18 (depends on 12) The system of claim 12, wherein the voice conversion model comprises a non-parallel voice conversion model. ### Claim 19 (depends on 12) The system of claim 12, wherein the ASR encoder was previously trained on ASR loss for a speech recognition task. ### Claim 20 (depends on 19) The system of claim 19, wherein the parameters of the pretrained ASR encoder remain fixed while training the VQ-VAE layer and the decoder of the voice conversion model. ### Claim 21 (depends on 12) The system of claim 12, wherein the VQ-VAE layer is trained using a VQ loss based on the latent representation of linguistic content generated for each timestep, the VQ loss encouraging the VQ-VAE layer to minimize a distance between an output and a nearest codebook. ### Claim 22 (depends on 12) The system of claim 12, wherein the decoder is configured to: receive, as input, the latent representation of linguistic content for the input speech and the speaker embedding; and generate, as output, the output speech comprising a reconstruction of the input speech.