Music enhancement systems

A system comprising:

• a mel spectrogram module implemented by one or more processing devices to:
• receive input data describing a recorded acoustic waveform of a musical instrument;
• and
• represent the recorded acoustic waveform of the musical instrument as an input mel spectrogram;
• a translation module implemented by the one or more processing devices to generate an enhanced mel spectrogram by processing the input mel spectrogram using a first machine learning model, the first machine learning model including a conditional generative adversarial network trained on a first type of training data to generate enhanced mel spectrograms based on input mel spectrograms, the first type of training data including pairs of recorded acoustic waveforms and perturbed acoustic waveforms that are generated by modifying the recorded acoustic waveforms, the pairs of recorded acoustic waveforms being used to generate high-quality mel spectrograms and low quality spectrograms for training the first machine learning model to generate the enhanced mel spectrograms;
• and
• a vocoding module implemented by the one or more processing devices to generate an acoustic waveform of the musical instrument using a second machine learning model trained to generate acoustic waveforms based on mel spectrograms, the second machine learning model is a denoising diffusion probabilistic model trained on a second type of training data, which is generated by iteratively adding Gaussian noise to the recorded acoustic waveforms used for generating the first type of training data, the second type of training data further including mel spectrograms computed for the recorded acoustic waveforms, the second machine learning model trained to estimate reverse transition distributions from adding the Gaussian noise to the recorded acoustic waveforms conditioned on the mel spectrograms computed for the recorded acoustic waveforms, the acoustic waveform of the musical instrument does not include an acoustic artifact that is included in the recorded acoustic waveform of the musical instrument.

A non-transitory computer-readable storage medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:

• receiving input data describing a recorded acoustic waveform of a musical instrument that includes an acoustic artifact;
• representing the recorded acoustic waveform of the musical instrument as an input mel spectrogram;
• generating an enhanced mel spectrogram by processing the input mel spectrogram using a first machine learning model, the first machine learning model including a conditional generative adversarial network trained on a first type of training data to generate enhanced mel spectrograms based on input mel spectrograms, the first type of training data including pairs of recorded acoustic waveforms and perturbed acoustic waveforms that are generated by modifying the recorded acoustic waveforms, the pairs of recorded acoustic waveforms being used to generate a high-quality mel spectrograms and low quality spectrograms for training the first machine learning model to generate the enhanced mel spectrograms;
• and
• generate an acoustic waveform of the musical instrument that does not include the acoustic artifact by processing the enhanced mel spectrogram using a second machine learning model trained to generate acoustic waveforms based on mel spectrograms, the second machine learning model is a denoising diffusion probabilistic model trained on a second type of training data, which is generated by iteratively adding Gaussian noise to the recorded acoustic waveforms used for generating the first type of training data, the second type of training data further including mel spectrograms computed for the recorded acoustic waveforms, the second machine learning model trained to estimate reverse transition distributions from adding the Gaussian noise to the recorded acoustic waveforms conditioned on the mel spectrograms computed for the recorded acoustic waveforms.