# US 2012/0253794 A1 · Voice conversion method and system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2012/0253794 A1
- Document ID: US20120253794A1
- Status: Published application
- Kind: A1
- Application number: 13/217,628
- Filed: 2011-08-25
- Published: 2012-10-04
- Earliest claimed: 2011-03-29
- Assignees: Kabushiki Kaisha Toshiba
- Inventors: Byung Ha CHUN; Mark John Francis GALES
- Web page: https://patentgrep.com/patent/US20120253794A1

## CPC

- G10L 21/003: Changing voice quality, e.g. pitch or formants
- G10L 21/007: characterised by the process used
- G10L 13/033: Voice editing, e.g. manipulating the voice of the synthesiser
- G10L 2021/0135: {Voice conversion or morphing}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A method of converting speech from the characteristics of a first voice to the characteristics of a second voice, the method comprising: receiving a speech input from a first voice, dividing said speech input into a plurality of frames; mapping the speech from the first voice to a second voice; and outputting the speech in the second voice, wherein mapping the speech from the first voice to the second voice comprises, deriving kernels demonstrating the similarity between speech features derived from the frames of the speech input from the first voice and stored frames of training data for said first voice, the training data corresponding to different text to that of the speech input and wherein the mapping step uses a plurality of kernels derived for each frame of input speech with a plurality of stored frames of training data of the first voice.

## Claims

### Claim 1

A method of converting speech from the characteristics of a first voice to the characteristics of a second voice, the method comprising:
receiving a speech input from a first voice, dividing said speech input into a plurality of frames;
mapping the speech from the first voice to a second voice; and
outputting the speech in the second voice,
wherein mapping the speech from the first voice to the second voice comprises, deriving kernels demonstrating the similarity between speech features derived from the frames of the speech input from the first voice and stored frames of training data for said first voice, the training data corresponding to different text to that of the speech input and wherein the mapping step uses a plurality of kernels derived for each frame of input speech with a plurality of stored frames of training data of the first voice.

### Claim 2 (depends on 1)

A method according to claim 1, wherein kernels are derived for both static and dynamic speech features.

### Claim 3 (depends on 1)

A method according to claim 1, wherein the speech to be output is determined according to a Gaussian Process predictive distribution:
p(yt|xt,x*,y*,)=(μ(xt),Σ(xt)),
where yt is the speech vector for frame t to be output, xt is the speech vector for the input speech for frame t, x*, y* is {x1*,y1*}, . . . , {xN*, yN*}, where xt* is the t-th frame of training data for the first voice and yt* is the t-th frame of training data for the second voice, M denotes the model, μ(xt) and Σ(xt) are the mean and variance of the predictive distribution for given xt.

### Claim 4 (depends on 3)

A method according to claim 3, wherein
μ

(
x
t
)
=
m

(
x
t
)
+
k
t
T

[
K
*
+
σ
2

I
]
-
1

(
y
*
-
μ
*
)
,

∑
(
x
t
)
=
k

(
x
t
,
x
t
)
+
σ
2
-
k
t
T

{
K
*
+
σ
2

I
]
-
1

k
t
,

where
μ
*
=
[
m

(
x
1
*
)


m

(
x
2
*
)


…


m

(
x
N
*
)
]
T
K
*
=
[
k

(
x
1
*
,
x
1
*
)
k

(
x
1
*
,
x
2
*
)
…
k

(
x
1
*
,
x
N
*
)
k

(
x
2
*
,
x
1
*
)
k

(
x
2
*
,
x
2
*
)
…
k

(
x
2
*
,
x
N
*
)
⋮
⋮
…
⋮
k

(
x
N
*
,
x
1
*
)
k

(
x
N
*
,
x
2
*
)
…
k

(
x
N
*
,
x
N
*
)
]
k
t
=
[
k

(
x
1
*
,
x
t
)


k

(
x
2
*
,
x
t
)


…


k

(
x
N
*
,
x
t
)
]
T
and σ is a parameter to be trained, m(xt) is a mean function and k(xt, xt′) is a kernel function representing the similarity between xt and xt′.

### Claim 5 (depends on 4)

A method according to claim 4, wherein the kernel function is isotropic.

### Claim 6 (depends on 4)

A method according to claim 4, wherein the kernel function is parameter free.

### Claim 7 (depends on 4)

A method according to claim 4, wherein the mean function is of the form:
m(xt)=axt+b

### Claim 8 (depends on 1)

A method according to claim 1, wherein the speech features are represented by vectors in an acoustic space and said acoustic space is partitioned for the training data such that a cluster of training data represents each part of the partitioned acoustic space, wherein during mapping, a frame of input speech is compared with the stored frames of training data for the first voice which have been assigned to the same cluster as the frame of input speech.

### Claim 9 (depends on 8)

A method according to claim 8, wherein two types of clusters are used, hard clusters and soft clusters, wherein in said hard clusters the boundary between adjacent clusters is hard so that there is no overlap between clusters and said soft clusters extend beyond the boundary of the hard clusters so that there is overlap between adjacent soft clusters, said frame of input speech being assigned to a cluster on the basis of the hard clusters.

### Claim 10 (depends on 9)

A method according to claim 9, wherein the frame of input speech which has been assigned to a cluster on the basis of hard clusters, is then compared with data from the extended soft cluster.

### Claim 11 (depends on 3)

A method according to claim 3, further comprising receiving training data for a first voice and a second voice.

### Claim 12 (depends on 11)

A method according to claim 11, further comprising training hyper-parameters from the training data.

### Claim 13 (depends on 1)

A method according to claim 1, wherein the first voice is a synthetic voice.

### Claim 14 (depends on 1)

A method according to claim 1, wherein the first voice comprises non-larynx excitations.

### Claim 15 (depends on 1)

A carrier medium carrying computer readable instructions for controlling the computer to carry out the method of claim 1.

### Claim 16

A system for converting speech from the characteristics of a first voice to the characteristics of a second voice, the system comprising:
a receiver for receiving a speech input from a first voice;
a processor configured to:
divide said speech input into a plurality of frames; and
map the speech from the first voice to a second voice,
the system further comprising an output to output the speech in the second voice,
wherein to map the speech from the first voice to the second voice, the processor is further adapted to derive kernels demonstrating the similarity between speech features derived from the frames of the speech input from the first voice and stored frames of training data for said first voice, the training data corresponding to different text to that of the speech input, the processor using a plurality of kernels derived for each frame of input speech with a plurality of stored frames of training data of the first voice.
