# US 12,001,950 B2 · Generative adversarial network based audio restoration > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,001,950 B2 - Document ID: US12001950B2 - Status: Granted - Kind: B2 - Application number: 16/299,828 - Filed: 2019-03-12 - Published: 2020-09-17 - Granted: 2024-06-04 - Earliest claimed: 2019-03-12 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Yang Zhang; Chuang Gan - Web page: https://patentgrep.com/patent/US12001950B2 ## CPC - G06N 3/08: Learning methods - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0475: Generative networks - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06N 3/09: Supervised learning - G06N 3/094: Adversarial learning - G10L 21/0208: Noise filtering - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Mechanisms are provided for implementing a generative adversarial network (GAN) based restoration system. A first neural network of a generator of the GAN based restoration system is trained to generate an artificial audio spectrogram having a target damage characteristic based on an input audio spectrogram and a target damage vector. An original audio recording spectrogram is input to the trained generator, where the original audio recording spectrogram corresponds to an original audio recording and an input target damage vector. The trained generator processes the original audio recording spectrogram to generate an artificial audio recording spectrogram having a level of damage corresponding to the input target damage vector. A spectrogram inversion module converts the artificial audio recording spectrogram to an artificial audio recording waveform output. ## Claims ### Claim 1 A method, in a data processing system comprising a processor and a memory, the memory comprising instructions which are executed by the processor to specifically configure the processor to implement a generative adversarial network (GAN) based restoration system, the method comprising: training, by the GAN based restoration system, a first neural network of a generator of the GAN based restoration system, to generate an artificial audio spectrogram having a target damage characteristic based on an input audio spectrogram and a target damage vector, thereby generating a trained generator; inputting, by the GAN based restoration system to the trained generator, an original audio recording spectrogram corresponding to an original audio recording and an input target damage vector; processing, by the trained generator, the original audio recording spectrogram to generate an artificial audio recording spectrogram having a level of damage corresponding to the input target damage vector; and converting, by a spectrogram inversion module of the GAN based restoration system, the artificial audio recording spectrogram to an artificial audio recording waveform output. ### Claim 2 (depends on 1) The method of claim 1, wherein the method further comprises: training, by the GAN based restoration system, a second neural network of a discriminator of the GAN based restoration system, to discriminate between artificial audio spectrograms and true audio spectrograms, and thereby generate a trained discriminator. ### Claim 3 (depends on 2) The method of claim 2, wherein training the first neural network comprises training the first neural network to generate an artificial audio spectrogram that causes the trained discriminator to misclassify the artificial audio spectrogram to be a true audio spectrogram. ### Claim 4 (depends on 2) The method of claim 2, wherein training the second neural network comprises iteratively training the second neural network to minimize a discriminator adversarial loss that represents the degree to which the discriminator can correctly recognize an input audio spectrogram as artificial, and wherein training the first neural network comprises iteratively training the first neural network to minimize a combination of both a generator adversarial loss that represents the degree to which the discriminator can recognize the artificial audio spectrogram output generated by the generator as artificial, and an identity loss that represents a self-to-self conversion to self. ### Claim 5 (depends on 1) The method of claim 1, wherein inputting the original audio recording spectrogram comprises processing, by a pre-processing module of the GAN based restoration system, an original audio waveform data structure representing the original audio recording comprising audio imperfections due to physical damage to a medium on which the original audio recording is recorded, to generate the original audio spectrogram and the input target damage vector based on the audio imperfections. ### Claim 6 (depends on 5) The method of claim 5, wherein the input target damage vector specifies one or more damage characteristics that represent an amount of physical damage to a target medium different from the physical damage to the medium on which the original audio recording is recorded. ### Claim 7 (depends on 1) The method of claim 1, wherein the input target damage vector is generated by a damage encoder of the GAN based restoration system, based on an input date label, wherein the damage encoder is trained to correlate a date label with a damage embedding represented as the input target damage vector. ### Claim 8 (depends on 1) The method of claim 1, wherein converting the artificial audio recording spectrogram to the artificial audio recording waveform output comprises introducing, by the spectrogram inversion module, a phase into the artificial audio spectrogram to thereby generate the artificial audio waveform data structure. ### Claim 9 (depends on 1) The method of claim 1, wherein the input target damage vector corresponds to a target damage level of a restored audio recording having a level of damage less than the original audio recording has, and the method further comprises outputting, by the GAN based restoration system, the artificial audio waveform data structure as a restored version of the original audio recording to one of a storage system for later playback, or to an audio output device for output as an audio output. ### Claim 10 (depends on 1) The method of claim 1, wherein the input target damage vector corresponds to a target damage level of a retro audio recording having a level of damage greater than the original audio recording has, and the method further comprises outputting, by the GAN based restoration system, the artificial audio waveform data structure as a retro version of the original audio recording to one of a storage system for later playback, or to an audio output device for output as an audio output. ### Claim 11 A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to implement a generative adversarial network (GAN) based restoration system that operates to: train a first neural network, of a generator of the GAN based restoration system, to generate an artificial audio spectrogram having a target damage characteristic based on an input audio spectrogram and a target damage vector, thereby generating a trained generator; input, to the trained generator, an original audio recording spectrogram corresponding to an original audio recording and an input target damage vector; process, by the trained generator, the original audio recording spectrogram to generate an artificial audio recording spectrogram having a level of damage corresponding to the input target damage vector; and convert, by a spectrogram inversion module of the GAN based restoration system, the artificial audio recording spectrogram to an artificial audio recording waveform output. ### Claim 12 (depends on 11) The computer program product of claim 11, wherein the computer readable program further causes the GAN based restoration system to: train a second neural network of a discriminator, of the GAN based restoration system, to discriminate between artificial audio spectrograms and true audio spectrograms, and thereby generate a trained discriminator. ### Claim 13 (depends on 12) The computer program product of claim 12, wherein training the first neural network of the generator comprises training the first neural network to generate an artificial audio spectrogram that causes the trained discriminator to misclassify the artificial audio spectrogram to be a true audio spectrogram. ### Claim 14 (depends on 12) The computer program product of claim 12, wherein training the second neural network comprises iteratively training the second neural network to minimize a discriminator adversarial loss that represents the degree to which the discriminator can correctly recognize an input audio spectrogram as artificial, and wherein training the first neural network comprises iteratively training the first neural network to minimize a combination of both a generator adversarial loss that represents the degree to which the discriminator can recognize the artificial audio spectrogram output generated by the generator as artificial, and an identity loss that represents a self-to-self conversion to self. ### Claim 15 (depends on 11) The computer program product of claim 11, wherein inputting the original audio recording spectrogram comprises processing, by a pre-processing module of the GAN based restoration system, an original audio waveform data structure representing the original audio recording comprising audio imperfections due to physical damage to a medium on which the original audio recording is recorded, to generate the original audio spectrogram and the input target damage vector based on the audio imperfections. ### Claim 16 (depends on 15) The computer program product of claim 15, wherein the input target damage vector specifies one or more damage characteristics that represent an amount of physical damage to a target medium different from the physical damage to the medium on which the original audio recording is recorded. ### Claim 17 (depends on 11) The computer program product of claim 11, wherein the input target damage vector is generated by a damage encoder of the GAN based restoration system, based on an input date label, wherein the damage encoder is trained to correlate a date label with a damage embedding represented as the input target damage vector. ### Claim 18 (depends on 11) The computer program product of claim 11, wherein converting the artificial audio recording spectrogram to the artificial audio recording waveform output comprises introducing, by the spectrogram inversion module, a phase into the artificial audio spectrogram to thereby generate the artificial audio waveform data structure. ### Claim 19 (depends on 11) The computer program product of claim 11, wherein the input target damage vector corresponds to a target damage level of a restored audio recording having a level of damage less than the original audio recording has, and the computer readable program further causes the GAN based restoration system to output the artificial audio waveform data structure as a restored version of the original audio recording to one of a storage system for later playback, or to an audio output device for output as an audio output. ### Claim 20 An apparatus comprising: a processor; and a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, causes the processor to implement a generative adversarial network (GAN) based restoration system that operates to: train a first neural network, of a generator of the GAN based restoration system, to generate an artificial audio spectrogram having a target damage characteristic based on an input audio spectrogram and a target damage vector, thereby generating a trained generator; input, to the trained generator, an original audio recording spectrogram corresponding to an original audio recording and an input target damage vector; process, by the trained generator, the original audio recording spectrogram to generate an artificial audio recording spectrogram having a level of damage corresponding to the input target damage vector; and convert, by a spectrogram inversion module of the GAN based restoration system, the artificial audio recording spectrogram to an artificial audio recording waveform output.