# US 11,114,085 B2 · Text-to-speech from media content item snippets

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,114,085 B2
- Document ID: US11114085B2
- Status: Granted
- Kind: B2
- Application number: 16/235,776
- Filed: 2018-12-28
- Published: 2020-07-02
- Granted: 2021-09-07
- Earliest claimed: 2018-12-28
- Assignees: Spotify AB
- Applicants: Spotify AB
- Inventors: Rohit Kumar; Henrik Lindström; Henriette Cramer; Sarah Mennicken; Sravana Reddy; Jennifer Thom-Santelli
- Web page: https://patentgrep.com/patent/US11114085B2

## CPC

- G10L 13/00: Speech synthesis; Text to speech systems
- G10L 13/06: Elementary speech units used in speech synthesisers; Concatenation rules
- G06F 16/685: {using automatically derived transcript of audio data, e.g. lyrics (speech recognition G10L15/00)}
- G10L 13/02: Methods for producing synthetic speech; Speech synthesisers
- G10L 13/04: Details of speech synthesis systems, e.g. synthesiser structure or memory management
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A text-to-speech engine creates audio output that includes synthesized speech and one or more media content item snippets. The input text is obtained and partitioned into text sets. A track having lyrics that match a part of one of the text sets is identified. The location of the track's audio that contains the lyric is extracted based on forced alignment data. The extracted audio is combined with synthesized speech corresponding to the remainder of the input text to form audio output.

## Claims

### Claim 1

A system providing text-to-speech functionality, comprising:
a forced alignment data store having stored thereon forced alignment data for tracks;
a combining engine configured to:
obtain input text;
portion the input text into a first text set, a second text set, and a third text set;
identify a first track having first track lyrics that include the first text set;
identify, using the forced alignment data, a first audio location of the first track corresponding to the first text set;
create a first audio snippet containing the first audio location;
identify a second track based on the second track having second track lyrics containing the third text set;
create a second audio snippet from the second track based on the third text set;
using a speech synthesizer, create a synthesized utterance based on the second text set;
combine the first audio snippet, the second audio snippet, and the synthesized utterance to form combined audio; and
provide an audio output that includes the combined audio.

### Claim 2 (depends on 1)

The system of claim 1, wherein the forced alignment data describes alignment between audio data of the tracks and lyrics data of the tracks.

### Claim 3 (depends on 2)

The system of claim 2, wherein the forced alignment data, for each of the tracks, describes lyrics data and time data of where lyrics occur in the audio data.

### Claim 4 (depends on 1)

The system of claim 1, wherein the identifying the second track is further based on audio characteristics of the first track and the second track.

### Claim 5 (depends on 4)

The system of claim 4, wherein identifying the second track includes:
selecting the second track based on similarities in musical style between the first track and the second track.

### Claim 6 (depends on 5)

Then system of claim 5, wherein the similarities in musical style between the first track and the second track is determined based on a distance between the first track and the second track in a vector space.

### Claim 7 (depends on 1)

The system of claim 1, further comprising a forced alignment engine for aligning input track audio and input track lyrics.

### Claim 8 (depends on 7)

The system of claim 7, wherein the forced alignment engine is configured to find a Viterbi path through the input track lyrics and the input track audio under an acoustic model.

### Claim 9 (depends on 7)

The system of claim 7, wherein the forced alignment engine is configured to align the input track lyrics and input track audio on a line-by-line basis.

### Claim 10 (depends on 7)

The system of claim 7, wherein the forced alignment engine is configured to:
receive a third track having track metadata;
receive third track lyrics associated with the third track;
select an acoustic model from an acoustic model data store based on the track metadata;
using the acoustic model, generate third-track forced alignment data that aligns audio data of the third track and the third track lyrics; and
add the third-track forced alignment data to the forced alignment data store.

### Claim 11 (depends on 1)

The system of claim 1, wherein the combining engine is further configured to:
determine a quality described in the input text,
wherein identifying the first track is based at least in part on the quality.

### Claim 12 (depends on 11)

The system of claim 11, wherein the quality is a music genre or music artist.

### Claim 13

A method comprising:
obtaining input text;
portioning the input text into a first text set, a second text set, and a third text set;
determine a quality described in the first text set,
identifying a first track based on:
the first track having first track lyrics that include the first text set; and
the first track having the quality;
identifying, using forced alignment data, a first audio location of the first track corresponding to the first text set;
creating a first audio snippet containing the first audio location;
identifying a second track based on the second track having second track lyrics containing the third text set based on similarities in musical style between the first track and the second track;
creating a second audio snippet from the second track based on the second text set;
using a speech synthesizer, creating a synthesized utterance based on the second text set;
combining the first audio snippet, the second audio snippet, and the synthesized utterance to form combined audio; and
providing an audio output that includes the combined audio.

### Claim 14

A system providing text-to-speech functionality, comprising:
a forced alignment engine for aligning input track audio and input track lyrics, wherein the forced alignment engine is configured to:
receive a third track having track metadata;
receive third track lyrics associated with the third track;
select an acoustic model from an acoustic model data store based on the track metadata;
using the acoustic model, generate third-track forced alignment data that aligns audio data of the third track and the third track lyrics; and
add the third-track forced alignment data to a forced alignment data store;
the forced alignment data store having stored thereon forced alignment data for tracks;
a combining engine configured to:
obtain input text;
portion the input text into a first text set and a second text set;
identify a first track having first track lyrics that include the first text set;
identify, using the forced alignment data, a first audio location of the first track corresponding to the first text set;
create a first audio snippet containing the first audio location;
using a speech synthesizer, create a synthesized utterance based on the second text set;
combine the first audio snippet and the synthesized utterance to form combined audio; and
provide an audio output that includes the combined audio.

### Claim 15 (depends on 14)

The system of claim 14, wherein the forced alignment data describes alignment between audio data of the tracks and lyrics data of the tracks.

### Claim 16 (depends on 15)

The system of claim 15, wherein the forced alignment data, for each of the tracks, describes lyrics data and time data of where lyrics occur in the audio data.

### Claim 17 (depends on 14)

The system of claim 14,
wherein the input text has a third text set; and
wherein the combining engine is further configured to:
identify a second track based on the second track having second track lyrics containing the third text set; and
create a second audio snippet from the second track based on the third text set,
wherein the combined audio further includes the second audio snippet.

### Claim 18 (depends on 17)

The system of claim 17, wherein the identifying the second track is further based on audio characteristics of the first track and the second track.

### Claim 19 (depends on 18)

The system of claim 18, wherein identifying the second track includes:
selecting the second track based on similarities in musical style between the first track and the second track.

### Claim 20 (depends on 19)

Then system of claim 19, wherein the similarities in musical style between the first track and the second track is determined based on a distance between the first track and the second track in a vector space.

### Claim 21 (depends on 14)

The system of claim 14, wherein the forced alignment engine is configured to find a Viterbi path through the input track lyrics and the input track audio under an acoustic model.

### Claim 22 (depends on 14)

The system of claim 14, wherein the forced alignment engine is configured to align the input track lyrics and input track audio on a line-by-line basis.

### Claim 23 (depends on 14)

The system of claim 14, wherein the combining engine is further configured to:
determine a quality described in the input text,
wherein identifying the first track is based at least in part on the quality.

### Claim 24 (depends on 23)

The system of claim 23, wherein the quality is a music genre or music artist.

### Claim 25 (depends on 13)

The method of claim 13, wherein the forced alignment data describes alignment between audio data and lyrics data of tracks.

### Claim 26 (depends on 25)

The method of claim 25, wherein the forced alignment data, for each of the tracks, describes lyrics data and time data of where lyrics occur in the audio data.

### Claim 27 (depends on 13)

The method of claim 13, wherein the identifying the second track is further based on audio characteristics of the first track and the second track.

### Claim 28 (depends on 27)

The method of claim 27, wherein identifying the second track includes:
selecting the second track based on similarities in musical style between the first track and the second track.

### Claim 29 (depends on 28)

Then method of claim 28, wherein the similarities in musical style between the first track and the second track is determined based on a distance between the first track and the second track in a vector space.

### Claim 30 (depends on 13)

The method of claim 13, further comprising:
using a forced alignment engine to find a Viterbi path through input track lyrics and input track audio under an acoustic model.

### Claim 31 (depends on 13)

The method of claim 13, further comprising:
using a forced alignment engine to align input track lyrics and input track audio on a line-by-line basis.

### Claim 32 (depends on 13)

The method of claim 13, further comprising:
with a forced alignment engine:
receiving a third track having track metadata;
receiving third track lyrics associated with the third track;
selecting an acoustic model from an acoustic model data store based on the track metadata;
with the acoustic model, generating third-track forced alignment data that aligns audio data of the third track and the third track lyrics; and
adding the third-track forced alignment data to a forced alignment data store.

### Claim 33 (depends on 13)

The method of claim 13, further comprising:
determining a quality described in the input text,
wherein identifying the first track is based at least in part on the quality.

### Claim 34 (depends on 33)

The method of claim 33, wherein the quality is a music genre or music artist.

### Claim 35

A method comprising:
with a forced alignment engine for aligning input track audio and input track lyrics:
receiving a third track having track metadata;
receiving third track lyrics associated with the third track;
selecting an acoustic model from an acoustic model data store based on the track metadata;
using the acoustic model, generating third-track forced alignment data that aligns audio data of the third track and the third track lyrics; and
adding the third-track forced alignment data to the forced alignment data store obtaining input text;
portioning the input text into a first text set a second text set;
determine a quality described in the first text set,
identifying a first track based on:
the first track having first track lyrics that include the first text set; and
the first track having the quality;
identifying, using forced alignment data, a first audio location of the first track corresponding to the first text set;
creating a first audio snippet containing the first audio location;
using a speech synthesizer, creating a synthesized utterance based on the second text set;
combining the first audio snippet and the synthesized utterance to form combined audio; and
providing an audio output that includes the combined audio.

### Claim 36 (depends on 35)

The method of claim 35, wherein the forced alignment data describes alignment between audio data and lyrics data of tracks.

### Claim 37 (depends on 36)

The method of claim 36, wherein the forced alignment data, for each of the tracks, describes lyrics data and time data of where lyrics occur in the audio data.

### Claim 38

The method of 35, further comprising:
identifying a second track based on audio characteristics of the first track and the second track,
wherein the combined audio includes a second audio snipped from the second track.

### Claim 39 (depends on 38)

The method of claim 38, wherein identifying the second track includes:
selecting the second track based on similarities in musical style between the first track and the second track.

### Claim 40 (depends on 39)

Then method of claim 39, wherein the similarities in musical style between the first track and the second track is determined based on a distance between the first track and the second track in a vector space.

### Claim 41 (depends on 35)

The method of claim 35, further comprising:
with the forced alignment engine, finding a Viterbi path through input track lyrics and input track audio under an acoustic model.

### Claim 42 (depends on 35)

The method of claim 35, further comprising:
with the forced alignment engine, aligning input track lyrics and input track audio on a line-by-line basis.

### Claim 43 (depends on 35)

The method of claim 35, further comprising:
determining a quality described in the input text,
wherein identifying the first track is based at least in part on the quality.

### Claim 44 (depends on 43)

The method of claim 43, wherein the quality is a music genre or music artist.