# US 2021/0241753 A1 · Text-to-speech from media content item snippets

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241753 A1
- Document ID: US20210241753A1
- Status: Published application
- Kind: A1
- Application number: 17/146,804
- Filed: 2021-01-12
- Published: 2021-08-05
- Earliest claimed: 2018-12-28
- Assignees: Spotify AB
- Applicants: Spotify AB
- Inventors: Rohit Kumar; Henrik Lindström; Henriette Cramer; Sarah Mennicken; Sravana Reddy; Jennifer Thom-Santelli
- Web page: https://patentgrep.com/patent/US20210241753A1

## CPC

- G10L 13/00: Speech synthesis; Text to speech systems
- G10L 13/06: Elementary speech units used in speech synthesisers; Concatenation rules
- G06F 16/685: {using automatically derived transcript of audio data, e.g. lyrics (speech recognition G10L15/00)}
- G10L 13/02: Methods for producing synthetic speech; Speech synthesisers
- G10L 13/04: Details of speech synthesis systems, e.g. synthesiser structure or memory management
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A text-to-speech engine creates audio output that includes synthesized speech and one or more media content item snippets. The input text is obtained and partitioned into text sets. A track having lyrics that match a part of one of the text sets is identified. The location of the track's audio that contains the lyric is extracted based on forced alignment data. The extracted audio is combined with synthesized speech corresponding to the remainder of the input text to form audio output.

## Claims

### Claim at position 1

1-15. (canceled)

### Claim 16

A system providing text-to-speech functionality, comprising:
a text-to-speech engine configured to:
identify, from input text, any one of (i) a plurality of words, (ii) a plurality of phrases, or (iii) a combination of (i) and (ii), suited to being converted into audio output;
identify from a plurality of tracks, at least one snippet having lyrics corresponding at least one of the plurality of words or the plurality of phrases suited to being matched to audio output corresponding to a snippet of a track of the plurality of tracks;
use a speech synthesizer to create a synthesized utterance of at least one of the plurality of words or plurality of phrases, thereby generating a synthesized utterance;
combine the at least one audio snippet and the synthesized utterance to form combined audio; and
provide an audio output that includes the combined audio.

### Claim 17 (depends on 16)

The system according to claim 16, further comprising:
a forced alignment data store having stored thereon forced alignment data for the plurality of tracks.

### Claim 18 (depends on 17)

The system of claim 17, wherein the forced alignment data describes alignment between audio data of the tracks and lyrics data of the tracks.

### Claim 19 (depends on 17)

The system of claim 17, wherein the forced alignment data, for each of the tracks, describes lyrics data and time data of where lyrics occur in the audio data.

### Claim 20 (depends on 17)

The system of claim 17, further comprising:
a combining engine configured to:
portion the input text into a first text set and a second text set;
identify a first track of the tracks, having first track lyrics that include the first text set;
identify, using the forced alignment data, a first audio location of first track audio data of the first track corresponding to the first text set;
create a first audio snippet containing the first audio location;
using the speech synthesizer, create the synthesized utterance based on the second text set; and
combine the first audio snippet and the synthesized utterance to form the combined audio.

### Claim 21 (depends on 20)

The system of claim 20, the combining engine further operable to:
determine a suitability of the first audio snippet, including by determining how clearly the lyrics of the first audio snippet can be heard.

### Claim 22 (depends on 20)

The system of claim 20,
wherein the input text has a third text set; and
wherein the combining engine is further configured to:
identify a second track based on the second track having second track lyrics containing the third text set; and
create a second audio snippet from the second track based on the third text set,
wherein the combined audio further includes the second audio snippet.

### Claim 23 (depends on 22)

The system of claim 22, wherein the identifying the second track is further based on audio characteristics of the first track and the second track.

### Claim 24 (depends on 23)

The system of claim 23, wherein identifying the second track includes:
selecting the second track based on similarities in musical style between the first track and the second track.

### Claim 25 (depends on 24)

The system of claim 24, wherein the similarities in musical style between the first track and the second track is determined based on a distance between the first track and the second track in a vector space.

### Claim 26 (depends on 16)

The system of claim 16, further comprising a forced alignment engine for aligning input track audio and input track lyrics.

### Claim 27 (depends on 26)

The system of claim 26, wherein the forced alignment engine is configured to find a Viterbi path through the input track lyrics and the input track audio under an acoustic model.

### Claim 28 (depends on 26)

The system of claim 26, wherein the forced alignment engine is configured to align the input track lyrics and input track audio on a line-by-line basis.

### Claim 29 (depends on 17)

The system of claim 17, further comprising:
a forced alignment engine is configured to:
receive a track of the plurality of tracks having track metadata;
receive track lyrics associated with the track;
select an acoustic model from an acoustic model data store based on the track metadata;
using the acoustic model, generate track forced alignment data that aligns audio data of the track and the track lyrics; and
add the track forced alignment data to the forced alignment data store.

### Claim 30 (depends on 20)

The system of claim 20, wherein the combining engine is further configured to:
determine a quality described in the input text,
wherein identifying the first track is based at least in part on the quality.

### Claim 31 (depends on 31)

The system of claim 31, wherein the quality is a music genre or music artist.

### Claim 32

A method for providing text-to-speech functionality, comprising:
identifying, from input text, any one of (i) a plurality of words, (ii) a plurality of phrases, or (iii) a combination of (i) and (ii), suited to being converted into audio output;
identifying from a plurality of tracks, at least one snippet having lyrics corresponding at least one of the plurality of words or the plurality of phrases suited to being matched to audio output corresponding to a snippet of a track of the plurality of tracks;
creating a synthesized utterance of at least one of the plurality of words or plurality of phrases, thereby generating a synthesized utterance;
combining the at least one audio snippet and the synthesized utterance to form combined audio; and
providing an audio output that includes the combined audio.

### Claim 33 (depends on 32)

The method according to claim 32, further comprising:
storing, in a forced alignment data store, forced alignment data for the plurality of tracks.

### Claim 34 (depends on 33)

The method of claim 33, wherein the forced alignment data describes alignment between audio data of the tracks and lyrics data of the tracks.

### Claim 35 (depends on 33)

The method of claim 33, wherein the forced alignment data, for each of the tracks, describes lyrics data and time data of where lyrics occur in the audio data.

### Claim 36 (depends on 33)

The method of claim 33, further comprising:
portioning the input text into a first text set and a second text set;
identifying a first track of the tracks, having first track lyrics that include the first text set;
identifying, using the forced alignment data, a first audio location of first track audio data of the first track corresponding to the first text set;
creating a first audio snippet containing the first audio location;
creating the synthesized utterance based on the second text set; and
combining the first audio snippet and the synthesized utterance to form the combined audio.

### Claim 37 (depends on 36)

The method of claim 36, further comprising:
determining a suitability of the first audio snippet, including by determining how clearly the lyrics of the first audio snippet can be heard.