# US 2012/0253781 A1 · Frame mapping approach for cross-lingual voice transformation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2012/0253781 A1
- Document ID: US20120253781A1
- Status: Published application
- Kind: A1
- Application number: 13/079,760
- Filed: 2011-04-04
- Published: 2012-10-04
- Earliest claimed: 2011-04-04
- Assignees: Microsoft Corporation
- Inventors: Yao Qian; Frank Kao-Ping Soong
- Web page: https://patentgrep.com/patent/US20120253781A1

## CPC

- G10L 21/003: Changing voice quality, e.g. pitch or formants
- G10L 2021/0135: {Voice conversion or morphing}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Frame mapping-based cross-lingual voice transformation may transform a target speech corpus in a particular language into a transformed target speech corpus that remains recognizable, and has the voice characteristics of a target speaker that provided the target speech corpus. A formant-based frequency warping is performed on the fundamental frequencies and the linear predictive coding (LPC) spectrums of source speech waveforms in a first language to produce transformed fundamental frequencies and transformed LPC spectrums. The transformed fundamental frequencies and the transformed LPC spectrums are then used to generate warped parameter trajectories. The warped parameter trajectories are further used to transform the target speech waveforms in the second language to produce transformed target speech waveform with voice characteristics of the first language that nevertheless retain at least some voice characteristics of the target speaker.

## Claims

### Claim 1

A computer-readable medium storing computer-executable instructions that, when executed, cause one or more processors to perform acts comprising:
performing formant-based frequency warping using fundamental frequencies and linear predictive coding (LPC) spectrums of source speech waveforms in a first language to produce transformed fundamental frequencies and transformed LPC spectrums;
generating warped parameter trajectories based at least on the transformed fundamental frequencies and the transformed LPC spectrums; and
producing transformed target speech waveforms with voice characteristics of the first language that retain at least some voice characteristics of a target speaker using the warped parameter trajectories and features from target speech waveforms of the target speaker in a second language.

### Claim 2 (depends on 1)

The computer-readable medium of claim 1, further comprising instructions that, when executed, cause the one or more processors to perform an act of generating synthesized speech for an input text using the transformed target speech waveforms.

### Claim 3 (depends on 2)

The computer-readable medium of claim 2, instructions that, when executed, cause the one or more processors to perform an act of estimating the LPC spectrums of the source speech waveforms using a Speech Transformation and Representation using Adaptive Interpolation of Weighted Spectrum (STRAIGHT) speech analysis.

### Claim 4 (depends on 1)

The computer-readable medium of claim 1, further comprising instructions that, when executed, cause the one or more processors to perform an act of extracting the fundamental frequencies of the source speech waveforms using pitch extraction.

### Claim 5 (depends on 1)

The computer-readable medium of claim 1, further comprising instructions that, when executed, cause the one or more processors to perform an act of obtaining linear spectrum pairs (LSPs) from the transformed LPC spectrums, wherein the generating further includes generating the warped parameter trajectories base at least on the transformed LPC spectrums and the LSPs that encapsulate the transformed LPC spectrums.

### Claim 6 (depends on 1)

The computer-readable medium of claim 1, further comprising instructions that, when executed, cause the one or more processors to perform an act of extracting the features that include fundamental frequencies, LSPs, and gains from the target speech waveforms.

### Claim 7 (depends on 1)

The computer-readable medium of claim 1, wherein the performing includes performing the formant-based frequency warping by:
aligning vowel segments embedded in a pair of speech utterances from a source speaker and a target speaker;
selecting stationary portions of a predefined length from the aligned vowel segments; and
defining a piece-wise linear interpolation function to warp the LPC spectrums based at least on a plurality of mapped formant pairs in the stationary portions, each mapped formant pair including a frequency anchor point for the source speaker and a frequency anchor point for the target speaker.

### Claim 8 (depends on 1)

The computer-readable medium of claim 1, wherein each frame of the transformed target speech waveforms in represented by a corresponding fundamental frequency, a corresponding LSP, and a corresponding gain, and wherein the producing the transformed target speech waveforms further includes:
selecting candidate frames of the target speech waveforms for a warped parameter trajectory based at least on distances between target frames in the warped parameter trajectory and the candidate frames; and
concatenating the selected candidate frames to form a target speech waveform.

### Claim 9 (depends on 1)

The computer-readable medium of claim 1, wherein the source speech waveforms are stored in a source speaker speech corpus, further comprising instructions that, when executed, cause the one or more processors to perform an act of storing the transformed target speech waveforms in a transformed target speaker speech corpus.

### Claim 10

A computer-implemented method, comprising:
under control of one or more computing systems configured with executable instructions,
performing formant-based frequency warping using fundamental frequencies and linear predictive coding (LPC) spectrums of source speech waveforms in a first language to produce transformed fundamental frequencies and transformed LPC spectrums;
generating warped parameter trajectories based at least on the transformed fundamental frequencies and the transformed LPC spectrums; and
producing transformed target speech waveforms with voice characteristics of the first language that retain at least some voice characteristics of a target speaker using the warped parameter trajectories and features from target speech waveforms of the target speaker in the second language;
training hidden markov models (HMMs) based at least on the transformed speech target waveforms; and
generating synthesized speech for an input text using the trained HMMs.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10, further comprising receiving input text from a text-to-speech application or a language translation application.

### Claim 12 (depends on 10)

The computer-implemented method of claim 10, further comprising:
estimating the LPC spectrums of the source speech waveforms using a Speech Transformation and Representation using Adaptive Interpolation of Weighted Spectrum (STRAIGHT) speech analysis;
extracting the fundamental frequencies of the source speech waveforms using pitch extraction; and
obtaining linear spectrum pairs (LSPs) from the transformed LPC spectrums,
wherein the generating further includes generating the warped parameter trajectories base at least on the transformed LPC spectrums and the LSPs.

### Claim 13 (depends on 10)

The computer-implemented method of claim 10, wherein the performing includes performing the formant-based frequency warping by:
aligning vowel segments embedded in a pair of speech utterances from a source speaker and a target speaker;
selecting stationary portions of a predefined length from the aligned vowel segments; and
defining a piece-wise linear interpolation function to warp the LPC spectrums based at least on a plurality of mapped formant pairs in the stationary portions, each mapped formant pair including a frequency anchor point for the source speaker and a frequency anchor point for the target speaker.

### Claim 14 (depends on 10)

The computer-implemented method of claim 10, further comprising extracting the features that include fundamental frequencies, LSPs, and gains from the target speech waveforms.

### Claim 15 (depends on 14)

The computer-implemented method of claim 14, wherein each frame of the transformed target speech waveforms in represented by a corresponding fundamental frequency, a corresponding LSP, and a corresponding gain, and wherein the producing the transformed target speech waveforms further includes:
selecting candidate frames of the target speech waveforms for a warped parameter trajectory based at least on distances between target frames in the warped parameter trajectory and the candidate frames; and
concatenating the selected candidate frames to form a target speech waveform.

### Claim 16

A system, comprising:
one or more processors; and
a memory that includes a plurality of computer-executable components, the plurality of computer-executable components comprising:
a frequency warping component to perform formant-based frequency warping using fundamental frequencies and linear predictive coding (LPC) spectrums of source speech waveforms in a first language to produce transformed fundamental frequencies and transformed LPC spectrums;
a LPC analysis component to obtain linear spectrum pairs (LSPs) from the transformed LPC spectrums;
a trajectory generation component to generate warped parameter trajectories based at least on the transformed fundamental frequencies and the transformed LPC spectrums; and
a trajectory tiling component to produce transformed target speech waveforms with voice characteristics of the first language that retain at least some voice characteristics of a target speaker using the warped parameter trajectories and features from target speech waveforms of the target speaker in the second language.

### Claim 17 (depends on 16)

The system of claim 16, further comprising:
a Speech Transformation and Representation using Adaptive Interpolation of Weighted Spectrum (STRAIGHT) analysis component to estimate the LPC spectrums of the source speech waveforms;
a pitch extraction component to extract fundamental frequencies of the source speech waveforms using pitch extraction; and
a feature extraction component to extract the features that include fundamental frequencies, LSPs, and gains from the target speech waveforms.

### Claim 18 (depends on 16)

The system of claim 16, further comprising a speech synthesis component to generating synthesized speech for an input text using hidden markov models (HMMs) trained with the transformed target speech waveforms.

### Claim 19 (depends on 16)

The system of claim 16, wherein the frequency warping component is to perform the formant-based frequency warping by:
aligning vowel segments embedded in a pair of speech utterances from a source speaker and a target speaker;
selecting stationary portions of a predefined length from the aligned vowel segments; and
defining a piece-wise linear interpolation function to warp the LPC spectrums based at least on a plurality of mapped formant pairs in the stationary portions, each mapped formant pair including a frequency anchor point for the source speaker and a frequency anchor point for the target speaker.

### Claim 20 (depends on 16)

The system of claim 16, wherein each frame of the transformed target speech waveforms in represented by a corresponding fundamental frequency, a corresponding LSP, and a corresponding gain, and wherein the trajectory tiling component is to produce the transformed target speech waveforms by:
selecting candidate frames of the target speech waveforms for a warped parameter trajectory based at least on distances between target frames in the warped parameter trajectory and the candidate frames; and
concatenating the selected candidate frames to form a target speech waveform.
