US 10,380,983 B2Granted

Machine learning to generate music from text

AssigneeGoogle Inc.

InventorDominik Roblek

Life of this patent

Estimating the term…

Loading patent…

Abstract

The present disclosure provides systems and methods that leverage one or more machine-learned models to generate music from text. In particular, a computing system can include a music generation model that is operable to extract one or more structural features from an input text. The one or more structural features can be indicative of a structure associated with the input text. The music generation model can generate a musical composition from the input text based at least in part on the one or more structural features. For example, the music generation model can generate a musical composition that exhibits a musical structure that mimics or otherwise corresponds to the structure associated with the input text. For example, the music generation model can include a machine-learned audio generation model. In such fashion, the systems and methods of the present disclosure can generate music that exhibits a globally consistent theme and/or structure.

Claims17

Dependent claims

A computer system to generate music from text, the system comprising:

a feature extractor configured to extract one or more structural features from an input text that comprises a plurality of portions, wherein the one or more structural features are indicative of a structure associated with the input text, and wherein the feature extractor is configured to extract at least one structural feature for each of the plurality of portions to generate a sequence of structural features from the input text;
a machine-learned deep neural network that is configured to receive each of the plurality of portions of the text and, in response, output a plurality of musical embeddings respectively for the plurality of portions, wherein the machine-learned deep neural network was trained on a training dataset that comprises a plurality of sets of lyrics from humanly-generated songs, wherein each set of lyrics is annotated with one or more music features descriptive of backing music associated with such set of lyrics in the corresponding humanly-generated song;
and
a machine-learned audio generation model configured to:
obtain the sequence of structural features from the feature extractor;
obtain the plurality of musical embeddings from the machine-learned deep neural network;
and
generate a musical composition from the input text based at least in part on the sequence of structural features and further based at least in part on the plurality of musical embeddings;
one or more processors;
and
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors cause the computer system to:
obtain the input text;
input the input text into the feature extractor;
receive the sequence of structural features as an output of the feature extractor;
input the input text into the machine-learned deep neural network;
receive the plurality of musical embeddings as an output of the machine-learned deep neural network;
input the sequence of structural features and the plurality of musical embeddings into the machine-learned audio generation model;
and
receive data descriptive of the musical composition as an output of the machine-learned audio generation model.

16 more elements

A computer-implemented method to generate music from text, the method comprising:

obtaining, by one or more computing devices, an input text that comprises a plurality of portions;
inputting, by the one or more computing devices, the input text into a feature extractor that is configured to extract at least one structural feature from each of the plurality portions of the input text;
receiving, by the one or more computing devices, a sequence of structural features as an output of the feature extractor, the sequence of structural features comprising the at least one structural feature extracted from each of the plurality portions of the input text;
inputting, by the one or more computing devices, the input text into a machine-learned deep neural network that was trained on a training dataset that comprises a plurality of sets of lyrics from humanly-generated songs, each set of lyrics annotated with one or more music features descriptive of backing music associated with such set of lyrics in the corresponding humanly-generated song;
receiving, by the one or more computing devices, a sequence of musical embeddings as an output of the machine-learned deep neural network, the sequence of musical embeddings comprising a plurality of musical embeddings respectively provided by the machine-learned deep neural network for the plurality of portions of the input text;
and
merging, by the one or more computing devices, the sequence of structural features with the sequence of musical embeddings in an interleaving fashion to form a merged sequence of features;
inputting, by the one or more computing devices, the merged sequence of features into a machine-learned audio generation model;
and
receiving, by the one or more computing devices, data descriptive of a musical composition generated by the machine-learned audio generation model based at least in part on the merged sequence of features.

7 more elements

One or more tangible, non-transitory computer-readable media that store instructions that, when executed by one or more processors, cause a computing system to:

obtain an input text that comprises a plurality of portions;
provide the input text to a machine-learned deep neural network that was trained on a training dataset that comprises a plurality of sets of lyrics from humanly-generated songs, each set of lyrics annotated with one or more music features descriptive of backing music associated with such set of lyrics in the corresponding humanly-generated song;
receive a sequence of musical embeddings as an output of the machine-learned deep neural network, the sequence of musical embeddings comprising a plurality of musical embeddings respectively provided by the machine-learned deep neural network for the plurality of portions of the input text;
and
input the sequence of musical embeddings into a machine-learned audio generation model;
and
receive data descriptive of a musical composition as an output of the machine-learned audio generation model.

4 more elements

Dependencies are inferred from the claim text.

Facts

Application: 15/394,895Filed 30 Dec 2016
Published: US 2018/0190249 A15 Jul 2018
Granted: 13 Aug 2019
Term ends: Not estimatedThe source records don’t support an estimate.

Classification19 codes

G10H 1/0025
Associated control or indicating means: automatic or semi-automatic music composition, e.g. producing random music, applying rules from music…
G06F 40/169
Editing, e.g. inserting or deleting: annotation, e.g. comment data or footnotes
G06F 40/30
Handling natural language data: semantic analysis
G06N 3/044
Architecture, e.g. interconnection topology: recurrent networks, e.g. Hopfield networks