# US 2008/0082333 A1 · Prosody Conversion

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2008/0082333 A1
- Document ID: US20080082333A1
- Status: Published application
- Kind: A1
- Application number: 11/536,701
- Filed: 2006-09-29
- Published: 2008-04-03
- Earliest claimed: 2006-09-29
- Assignees: Nokia Corporation
- Inventors: Jani K. Nurminen; Elina Helander
- Web page: https://patentgrep.com/patent/US20080082333A1

## CPC

- G10L 21/00: Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility (G10L19/00 takes precedence)
- G10L 13/04: Details of speech synthesis systems, e.g. synthesiser structure or memory management
- G10L 2021/0135: {Voice conversion or morphing}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A contour for a syllable (or other speech segment) in a voice undergoing conversion is transformed. The transform of that contour is then used to identify one or more source syllable transforms in a codebook. Information regarding the context and/or linguistic features of the contour being converted can also be compared to similar information in the codebook when identifying an appropriate source transform. Once a codebook source transform is selected, an inverse transformation is performed on a corresponding codebook target transform to yield an output contour. The corresponding codebook target transform represents a target voice version of the same syllable represented by the selected codebook source transform. The output contour may be further processed to improve conversion quality.

## Claims

### Claim 1

A method comprising:
(a) receiving data for a plurality of segments of a passage in a source voice, wherein the data for each segment of the plurality models a prosodic component of the source voice for that segment;
(b) identifying a target voice entry in a codebook for each of the source voice passage segments, wherein each of the identified target voice entries models a prosodic component of a target voice for a different segment of codebook training material, and wherein the codebook training material is substantially different from the passage; and
(c) generating a target voice version of the plurality of passage segments by altering the modeled source voice prosodic component for each segment to replicate the target voice prosodic component modeled by the target voice entry identified for that segment in (b).

### Claim 2 (depends on 1)

The method of claim 1, wherein operation (a) includes receiving data for one or more additional segments of the passage in a source voice, and wherein the method further comprises:
(d) generating a target voice version of each of the one or more additional source voice passage segments according to
x
i

(
n
)

|
MV
=
x
i
SRC

(
n
)
-
μ
SRC
σ
SRC
*
σ
TGT
+
μ
TGT
wherein
μSRC is a mean of all F0 values for source voice versions of segments in the codebook training material,
σSRC is a standard deviation of all F0 values for source voice versions of segments in the codebook training material,
μTGT is a mean of all F0 values for target voice versions of segments in the codebook training material,
σTGT is a standard deviation of all F0 values for target voice versions of segments in the codebook training material,
xiSRC(n) is a value for F0 at time n in an F0 contour for segment i of the additional segments, and
xi(n)|MV is a value for F0 at time n in an F0 contour for a target voice version of segment i of the additional segments.

### Claim 3 (depends on 1)

The method of claim 1, wherein
the codebook includes multiple source voice entries,
each of the multiple source voice entries models a prosodic component of the source voice for a different segment of the codebook training material,
each of the multiple source voice entries corresponds to a target voice entry modeling a prosodic component of the target voice for the segment of the codebook training material for which the corresponding source voice entry models the prosodic component of the source voice, and
operation (b) includes, for each source voice passage segment, identifying a target voice entry by comparing data for the source voice passage segment to one or more of the multiple source voice entries.

### Claim 4 (depends on 3)

The method of claim 3, wherein
each of the multiple source voice entries and its corresponding target voice entry includes a plurality of transform coefficients representing a contour for the modeled prosodic component, and
operation (b) includes, for each source voice passage segment, identifying a target voice entry by comparing transform coefficients representing a contour for the prosodic component of the source voice passage segment to the transform coefficients for one or more of the multiple source voice entries.

### Claim 5 (depends on 4)

The method of claim 4, wherein
each of the multiple source voice entries is associated with a different feature vector,
each of the associated feature vectors includes values of a set of linguistic features for the codebook training speech segment for which the associated source voice entry models the prosodic component of the source voice,
data for each of the source voice passage segments includes a feature vector that includes values of the set of linguistic features for that source voice passage segment, and
operation (b) includes, for each source voice passage segment,
(b1) identifying multiple candidate source voice entries based the transform coefficient comparisons, and
(b2) selecting the identified target voice entry based on a comparison of the feature vector for the source voice passage segment with each of the feature vectors associated with the multiple candidate source voice entries identified in (b1).

### Claim 6 (depends on 5)

The method of claim 5, wherein the selecting in operation (b2) is also based on comparison of a duration of the source voice passage segment with durations of each of the candidate source voice entries identified in (b1).

### Claim 7

A machine-readable medium having machine-executable instructions for performing a method comprising:
(a) receiving data for a plurality of segments of a passage in a source voice, wherein the data for each segment of the plurality models a prosodic component of the source voice for that segment;
(b) identifying a target voice entry in a codebook for each of the source voice passage segments, wherein each of the identified target voice entries models a prosodic component of a target voice for a different segment of codebook training material, and wherein the codebook training material is substantially different from the passage; and
(c) generating a target voice version of the plurality of passage segments by altering the modeled source voice prosodic component for each segment to replicate the target voice prosodic component modeled by the target voice entry identified for that segment in (b).

### Claim 8 (depends on 7)

The machine-readable medium of claim 7, wherein operation (a) includes receiving data for one or more additional segments of the passage in a source voice, and comprising additional machine-executable instructions for:
(d) generating a target voice version of each of the one or more additional source voice passage segments according to
x
i

(
n
)

|
MV
=
x
i
SRC

(
n
)
-
μ
SRC
σ
SRC
*
σ
TGT
+
μ
TGT
wherein
μSRC is a mean of all F0 values for source voice versions of segments in the codebook training material,
σSRC is a standard deviation of all F0 values for source voice versions of segments in the codebook training material,
μTGT is a mean of all F0 values for target voice versions of segments in the codebook training material,
σTGT is a standard deviation of all F0 values for target voice versions of segments in the codebook training material,
xiSRC(n) is a value for F0 at time n in an F0 contour for segment i of the additional segments, and
xi(n)|MV is a value for F0 at time n in an F0 contour for a target voice version of segment i of the additional segments.

### Claim 9 (depends on 8)

The machine-readable medium of claim 8, wherein the data for the passage segments in the source voice is generated by a text-to-speech system.

### Claim 10 (depends on 7)

The machine-readable medium of claim 7, wherein the modeled prosodic components are pitch contours.

### Claim 11 (depends on 7)

The machine-readable medium of claim 7, wherein
the codebook includes multiple source voice entries,
each of the multiple source voice entries models a prosodic component of the source voice for a different segment of the codebook training material,
each of the multiple source voice entries corresponds to a target voice entry modeling a prosodic component of the target voice for the segment of the codebook training material for which the corresponding source voice entry models the prosodic component of the source voice, and
operation (b) includes, for each source voice passage segment, identifying a target voice entry by comparing data for the source voice passage segment to one or more of the multiple source voice entries.

### Claim 12 (depends on 11)

The machine-readable medium of claim 11, wherein
each of the multiple source voice entries and its corresponding target voice entry includes a plurality of transform coefficients representing a contour for the modeled prosodic component, and
operation (b) includes, for each source voice passage segment, identifying a target voice entry by comparing transform coefficients representing a contour for the prosodic component of the source voice passage segment to the transform coefficients for one or more of the multiple source voice entries.

### Claim 13 (depends on 12)

The machine-readable medium of claim 12, wherein the transform is a discrete cosine transform.

### Claim 14 (depends on 12)

The machine-readable medium of claim 12, wherein
each of the multiple source voice entries is associated with a different feature vector,
each of the associated feature vectors includes values of a set of linguistic features for the codebook training speech segment for which the associated source voice entry models the prosodic component of the source voice,
data for each of the source voice passage segments includes a feature vector that includes values of the set of linguistic features for that source voice passage segment, and
operation (b) includes, for each source voice passage segment,
(b1) identifying multiple candidate source voice entries based the transform coefficient comparisons, and
(b2) selecting the identified target voice entry based on a comparison of the feature vector for the source voice passage segment with each of the feature vectors associated with the multiple candidate source voice entries identified in (b1).

### Claim 15 (depends on 14)

The machine-readable medium of claim 14, wherein the selecting in operation (b2) is also based on comparison of a duration of the source voice passage segment with durations of each of the candidate source voice entries identified in (b1).

### Claim 16 (depends on 12)

The machine-readable medium of claim 12, wherein operation (c) includes,
(c1) performing an inverse transform on the target voice entry identified for one of the source voice passage segments,
(c2) adjusting the result of (c1) according to
xiTGT(n)|a=xiTGT(n)+xiSRC(n)−zjSRC(n),
wherein xiTGT(n) is a value for pitch at time n and is the result of (c1), XiSRC(n) is a value for pitch at time n from a pitch contour for the source voice passage segment for which the inverse transform was performed in (c1), zjSRC(n) is a value for pitch at time n obtained from the inverse transform of the source voice entry corresponding to the identified target voice entry of (c1), and xiTGT(n)|a is an adjusted pitch value at time n.

### Claim 17 (depends on 16)

The machine-readable medium of claim 16, wherein operation (c) includes
(c3) further adjusting the result of (c2) according to
xiTGT(n)|a, μ=xiTGT(n)|a +xi(n)|MV,
wherein
x
i

(
n
)

|
MV
=
x
i
SRC

(
n
)
-
μ
SRC
σ
SRC
*
σ
TGT
+
μ
TGT
and wherein
μSRC is a mean of all F0 values for source voice versions of segments in the codebook training material,
σSRC is a standard deviation of all F0 values for source voice versions of segments in the codebook training material,
μTGT is a mean of all F0 values for target voice versions of segments in the codebook training material, and
σTGT is a standard deviation of all F0 values for target voice versions of segments in the codebook training material.

### Claim 18 (depends on 17)

The machine-readable medium of claim 17, wherein operation (c) includes
(c4) determining whether a boundary between the source voice passage segment for which the inverse transform was performed in (c1) and an adjacent source voice passage segment is continuous in voicing energy level, and
(c5) upon determining in (c4) that the boundary is continuous in voicing energy level, adding a bias value to the result of (c3) to preserve a continuous pitch level.

### Claim 19

A device, comprising:
one or more processors configured to perform a method, the method including
(a) receiving data for a plurality of segments of a passage in a source voice, wherein the data for each segment of the plurality models a prosodic component of the source voice for that segment,
(b) identifying a target voice entry in a codebook for each of the source voice passage segments, wherein each of the identified target voice entries models a prosodic component of a target voice for a different segment of codebook training material, and wherein the codebook training material is substantially different from the passage, and
(c) generating a target voice version of the plurality of passage segments by altering the modeled source voice prosodic component for each segment to replicate the target voice prosodic component modeled by the target voice entry identified for that segment in (b).

### Claim 20 (depends on 19)

The device of claim 19, wherein operation (a) includes receiving data for one or more additional segments of the passage in a source voice, and wherein the one or more processors are configured to generate a target voice version of each of the one or more additional source voice passage segments according to
x
i

(
n
)

|
MV
=
x
i
SRC

(
n
)
-
μ
SRC
σ
SRC
*
σ
TGT
+
μ
TGT
wherein
μSRC is a mean of all F0 values for source voice versions of segments in the codebook training material,
σSRC is a standard deviation of all F0 values for source voice versions of segments in the codebook training material,
μTGT is a mean of all F0 values for target voice versions of segments in the codebook training material,
σTGT is a standard deviation of all F0 values for target voice versions of segments in the codebook training material,
xiSRC(n) is a value for F0 at time n in an F0 contour for segment i of the additional segments, and
xi(n)|MV is a value for F0 at time n in an F0 contour for a target voice version of segment i of the additional segments.

### Claim 21 (depends on 20)

The device of claim 20, wherein the data for the passage segments in the source voice is generated by a text-to-speech system.

### Claim 22 (depends on 19)

The device of claim 19, wherein the modeled prosodic components are pitch contours.

### Claim 23 (depends on 19)

The device of claim 19, wherein
the codebook includes multiple source voice entries,
each of the multiple source voice entries models a prosodic component of the source voice for a different segment of the codebook training material,
each of the multiple source voice entries corresponds to a target voice entry modeling a prosodic component of the target voice for the segment of the codebook training material for which the corresponding source voice entry models the prosodic component of the source voice, and
operation (b) includes, for each source voice passage segment, identifying a target voice entry by comparing data for the source voice passage segment to one or more of the multiple source voice entries.

### Claim 24 (depends on 23)

The device of claim 23, wherein
each of the multiple source voice entries and its corresponding target voice entry includes a plurality of transform coefficients representing a contour for the modeled prosodic component, and
operation (b) includes, for each source voice passage segment, identifying a target voice entry by comparing transform coefficients representing a contour for the prosodic component of the source voice passage segment to the transform coefficients for one or more of the multiple source voice entries.

### Claim 25 (depends on 24)

The device of claim 24, wherein the transform is a discrete cosine transform.

### Claim 26 (depends on 24)

The device of claim 24, wherein
each of the multiple source voice entries is associated with a different feature vector,
each of the associated feature vectors includes values of a set of linguistic features for the codebook training speech segment for which the associated source voice entry models the prosodic component of the source voice,
data for each of the source voice passage segments includes a feature vector that includes values of the set of linguistic features for that source voice passage segment, and
operation (b) includes, for each source voice passage segment,
(b1) identifying multiple candidate source voice entries based the transform coefficient comparisons, and
(b2) selecting the identified target voice entry based on a comparison of the feature vector for the source voice passage segment with each of the feature vectors associated with the multiple candidate source voice entries identified in (b1).

### Claim 27 (depends on 26)

The device of claim 26, wherein the selecting in operation (b2) is also based on comparison of a duration of the source voice passage segment with durations of each of the candidate source voice entries identified in (b1).

### Claim 28 (depends on 24)

The device of claim 24, wherein operation (c) includes,
(c1) performing an inverse transform on the target voice entry identified for one of the source voice passage segments,
(c2) adjusting the result of (c1) according to
xiTGT(n)|a=xiTGT(n)+xiSRC(n)−zjSRC(n),
wherein xiTGT(n) is a value for pitch at time n and is the result of (c1), XiSRC(n) is a value for pitch at time n from a pitch contour for the source voice passage segment for which the inverse transform was performed in (c1), zjSRC(n) is a value for pitch at time n obtained from the inverse transform of the source voice entry corresponding to the identified target voice entry of (c1), and xiTGT(n)|a is an adjusted pitch value at time n.

### Claim 29 (depends on 28)

The device of claim 28, wherein operation (c) includes
(c3) further adjusting the result of (c2) according to
x
i
TGT

(
n
)


a
,
μ
=
x
i
TGT

(
n
)

|
a
+
x
i

(
n
)

|
MV
,

wherein
x
i

(
n
)

|
MV
=
x
i
SRC

(
n
)
-
μ
SRC
σ
SRC
*
σ
TGT
+
μ
TGT
and wherein
μSRC is a mean of all F0 values for source voice versions of segments in the codebook training material,
σSRC is a standard deviation of all F0 values for source voice versions of segments in the codebook training material,
μTGT is a mean of all F0 values for target voice versions of segments in the codebook training material, and
σTGT is a standard deviation of all F0 values for target voice versions of segments in the codebook training material.

### Claim 30 (depends on 29)

The device of claim 29, wherein operation (c) includes
(c4) determining whether a boundary between the source voice passage segment for which the inverse transform was performed in (c1) and an adjacent source voice passage segment is continuous in voicing energy level, and
(c5) upon determining in (c4) that the boundary is continuous in voicing energy level, adding a bias value to the result of (c3) to preserve a continuous pitch level.

### Claim 31 (depends on 19)

The device of claim 19, wherein the device is a mobile communication device.

### Claim 32 (depends on 19)

The device of claim 19, wherein the device is a computer.

### Claim 33

A device, comprising:
a voice converter, the voice converter including
means for receiving data for a plurality of segments of a passage in a source voice,
means for identifying target voice data entries in a codebook for segments of the source voice passage, and
means for generating a target voice version of the passage segments based on identified target voice data entries.

### Claim 34 (depends on 33)

The device of claim 33, wherein the identification means include means for comparing transformed representations of source passage pitch contours to transformed representations of codebook training material pitch contours.

### Claim 35 (depends on 33)

The device of claim 33, wherein the identification means include means for comparing feature vectors of source passage segments to feature vectors of codebook training material segments.
