# US 11,308,943 B2 · Systems and methods for aligning lyrics using a neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,943 B2
- Document ID: US11308943B2
- Status: Granted
- Kind: B2
- Application number: 16/569,372
- Filed: 2019-09-12
- Published: 2020-04-30
- Granted: 2022-04-19
- Earliest claimed: 2018-10-29
- Assignees: Spotify AB
- Applicants: Spotify AB
- Inventors: Daniel Stoller; Simon René Georges Durand; Sebastian Ewert
- Web page: https://patentgrep.com/patent/US11308943B2

## CPC

- G10L 15/183: using context dependencies, e.g. language models
- G06F 16/63: Querying
- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G10L 15/16: using artificial neural networks
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 2015/088: {Word spotting}
- G10L 2015/226: {using non-speech characteristics}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

An electronic device receives audio data for a media item. The electronic device generates, from the audio data, a plurality of samples, each sample having a predefined maximum length. The electronic device, using a neural network trained to predict character probabilities, generates a probability matrix of characters for a first portion of a first sample of the plurality of samples. The probability matrix includes character information, timing information, and respective probabilities of respective characters at respective times. The electronic device identifies, for the first portion of the first sample, a first sequence of characters based on the generated probability matrix.

## Claims

### Claim 1

A method, comprising:
at an electronic device having one or more processors and memory storing instructions for execution by the one or more processors:
receiving audio data for a media item;
generating, from the audio data, a plurality of samples, each sample having a predefined maximum length;
using a neural network trained to predict character probabilities, generating a plurality of probability matrices for the plurality of samples, including, for a respective sample of the plurality of samples, generating a probability matrix of characters for the respective sample of the plurality of samples, including:
downsampling at least a portion of the respective sample to reduce a dimension of the respective sample;
convolving an output of the downsampling of the respective sample; and
upsampling an output of the convolution of the first respective sample to increase the dimension of the respective sample;
wherein the probability matrix includes:
character information,
timing information, and
respective probabilities of respective characters at respective times;
concatenating a set of two or more of the plurality of probability matrices to create a single probability matrix, the single probability matrix including:
character information,
timing information, and
respective probabilities of respective characters at respective times; and
identifying a first sequence of characters using the single probability matrix.

### Claim 2 (depends on 1)

The method of claim 1, wherein identifying the first sequence of characters includes:
receiving, from an external source, lyrics corresponding to the media item; and
using the received lyrics and the single probability matrix, aligning characters in the first sequence of characters with the received lyrics corresponding to the media item.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
determining a set of lyrics based on the first sequence of characters; and
storing the set of lyrics in association with the media item.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
using a language model and at least a portion of the first sequence of characters, determining a first word in a first sample; and
determining, using the timing information that corresponds to the first sample, a time that corresponds to the first word.

### Claim 5 (depends on 1)

The method of claim 1, wherein the received audio data includes an extracted vocal track that has been separated from a media content item.

### Claim 6 (depends on 1)

The method of claim 1, wherein the received audio data is a polyphonic media content item.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
receiving, from a user, a request to search for a second sequence of characters within the media item;
in response to receiving the request to search for the second sequence of characters, performing a search of the first sequence of characters to determine whether at least a portion of the first sequence of characters matches the second sequence of characters; and
in accordance with a determination that at least a portion of the first sequence of characters matches the second sequence of characters, identifying timing information related to the portion that matches.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
identifying, from the first sequence of characters, one or more keywords associated with the media item.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
determining whether any of the one or more keywords corresponds to a defined set of words; and
in accordance with a determination that a first keyword of the one or more keywords corresponds to the defined set of words, performing an operation on a portion of the sample that corresponds to the first keyword.

### Claim 10

A first electronic device comprising:
one or more processors; and
memory storing instructions for execution by the one or more processors, the instructions including instructions for:
receiving audio data for a media item;
generating, from the audio data, a plurality of samples, each sample having a predefined maximum length;
using a neural network trained to predict character probabilities, generating a plurality of probability matrices for the plurality of samples, including, for a respective sample of the plurality of samples, generating a probability matrix of characters for the respective sample of the plurality of samples, including:
downsampling at least a portion of the respective sample to reduce a dimension of the respective sample;
convolving an output of the downsampling of the first respective sample; and
upsampling an output of the convolution of the first respective sample to increase the dimension of the respective sample;
wherein the probability matrix includes:
character information,
timing information, and
respective probabilities of respective characters at respective times;
concatenating a set of two or more of the plurality of probability matrices to create a single probability matrix, the single probability matrix including:
character information,
timing information, and
respective probabilities of respective characters at respective times; and
identifying a first sequence of characters using the single probability matrix.

### Claim 11

A non-transitory computer-readable storage medium storing instructions that, when executed by an electronic device, cause the electronic device to:
receive audio data for a media item;
generate, from the audio data, a plurality of samples, each sample having a predefined maximum length;
using a neural network trained to predict character probabilities, generate a plurality of probability matrices for the plurality of samples, including, for a respective sample of the plurality of samples, generating a probability matrix of characters for the respective sample of the plurality of samples, including:
downsampling at least a portion of the respective sample to reduce a dimension of the respective sample;
convolving an output of the downsampling of the respective sample; and
upsampling an output of the convolution of the respective sample to increase the dimension of the first respective sample;
wherein the probability matrix includes:
character information,
timing information, and
respective probabilities of respective characters at respective times;
concatenate a set of two or more of the plurality of probability matrices to create a single probability matrix, the single probability matrix including:
character information,
timing information, and
respective probabilities of respective characters at respective times; and
identify a first sequence of characters using the generated single probability matrix.

### Claim 12 (depends on 10)

The electronic device of claim 10, wherein identifying the first sequence of characters includes:
receiving, from an external source, lyrics corresponding to the media item; and
using the received lyrics and the single probability matrix, aligning characters in the first sequence of characters with the received lyrics corresponding to the media item.

### Claim 13 (depends on 10)

The electronic device of claim 10, the one or more programs further including instructions for:
determining a set of lyrics based on the first sequence of characters; and
storing the set of lyrics in association with the media item.

### Claim 14 (depends on 10)

The electronic device of claim 10, the one or more programs further including instructions for:
using a language model and at least a portion of the first sequence of characters, determining a first word in a first sample; and
determining, using the timing information that corresponds to the first sample, a time that corresponds to the first word.

### Claim 15 (depends on 10)

The electronic device of claim 10, wherein the received audio data includes an extracted vocal track that has been separated from a media content item.

### Claim 16 (depends on 10)

The electronic device of claim 10, wherein the received audio data is a polyphonic media content item.

### Claim 17 (depends on 10)

The electronic device of claim 10, the one or more programs further including instructions for:
receiving, from a user, a request to search for a second sequence of characters within the media item;
in response to receiving the request to search for the second sequence of characters, performing a search of the first sequence of characters to determine whether at least a portion of the first sequence of characters matches the second sequence of characters; and
in accordance with a determination that at least a portion of the first sequence of characters matches the second sequence of characters, identifying timing information related to the portion that matches.