Implementing automatic music audio transcription

A system of extracting a melody, comprising:

• at least one processor;
• and
• at least one memory communicatively coupled to the at least one processor and comprising computer-readable instructions that upon execution by the at least one processor cause the at least one processor to perform operations comprising:
• configuring a deep neural network model of implementing the automatic music transcription, wherein the deep neural network model comprises a spectral cross-attention sub-model configured to project a spectral representation of each time step t, denoted as St, into a set of latent arrays at the time step t, denoted as θth, h representing an h-th iteration, wherein the deep neutral network model comprises a plurality of latent transformers configured to perform self-attention on the set of latent arrays θth, and wherein the deep neural network model further comprises a set of temporal transformers configured to enable communications between any pairs of latent arrays θth at different time steps;
• augmenting training data by randomly mixing a plurality of types of dataset, wherein the plurality of types of dataset comprise a vocal dataset and an instrument dataset;
• and
• training the deep neural network model using the augmented training data, wherein the deep neural network model is trained to automatically transcribe music audio comprising vocal and multi-instrument tracks into a plurality of sets of MIDI data corresponding to each of the vocal and multi-instrument tracks.

A non-transitory computer-readable storage medium, storing computer-readable instructions that upon execution by a processor cause the processor to implement operations comprising:

• configuring a deep neural network model of implementing the automatic music transcription, wherein the deep neural network model comprises a spectral cross-attention sub-model configured to project a spectral representation of each time step t, denoted as St, into a set of latent arrays at the time step t, denoted as θth, h representing an h-th iteration, wherein the deep neutral network model comprises a plurality of latent transformers configured to perform self-attention on the set of latent arrays θth and wherein the deep neural network model further comprises a set of temporal transformers configured to enable communications between any pairs of latent arrays θth at different time steps;
• augmenting training data by randomly mixing a plurality of types of dataset, wherein the plurality of types of dataset comprise a vocal dataset and an instrument dataset;
• and
• training the deep neural network model using the augmented training data, wherein the deep neural network model is trained to automatically transcribe music audio comprising vocal and multi-instrument tracks into a plurality of sets of MIDI data corresponding to each of the vocal and multi-instrument tracks.