Audio processing of missing audio information

An audio processing method, comprising:

• obtaining to-be-processed audio data and frequency spectrum information of the to-be-processed audio data, the to-be-processed audio data including an audio missing segment, and the frequency spectrum information including frequency spectrum features of the to-be-processed audio data;
• and
• obtaining processed audio data based on modification of a portion of the to-be-processed audio data according to predicted audio data obtained based on the frequency spectrum features of the to-be-processed audio data through a generative adversarial network (GAN) model,
• wherein the GAN model is trained by:
• obtaining training audio data corresponding to a time interval in a time domain;
• obtaining target audio data based on the training audio data and a simulative packet loss adjustment, the target audio data corresponding to the time interval in the time domain and including an audio missing segment, context audio segments including a first valid audio segment preceding the audio missing segment and a second valid audio segment succeeding the audio missing segment, and one or more other valid audio segments;
• obtaining frequency spectrum information of the target audio data, the frequency spectrum information being in a frequency domain and based on transformation of the target audio data from the time domain to the frequency domain;
• obtaining compensated frequency spectrum information based on the frequency spectrum information of the target audio data through a generator of the GAN model, the generator including a plurality of compensation subunits, each one of the plurality of compensation subunits including an upsampling layer and a convolution layer;
• generating predicted audio data corresponding to the time interval in the time domain based on the compensated frequency spectrum information;
• obtaining compensated audio data based on modification of a portion of the target audio data according to a corresponding portion of the predicted audio data, the portion of the target audio data including the audio missing segment and a subset of the context audio segments;
• determining difference information between the compensated audio data and the training audio data through a discriminator of the GAN model;
• and
• updating the generator or the generator and the discriminator based on the difference information.

An apparatus of training a generative adversarial network (GAN) model, the apparatus comprising:

• processing circuitry configured to:
• obtain training audio data corresponding to a time interval in a time domain;
• obtain target audio data based on the training audio data and a simulative packet loss adjustment, the target audio data corresponding to the time interval in the time domain and including an audio missing segment, context audio segments including a first valid audio segment preceding the audio missing segment and a second valid audio segment succeeding the audio missing segment, and one or more other valid audio segments;
• obtain frequency spectrum information of the target audio data, the frequency spectrum information being in a frequency domain and based on transformation of the target audio data from the time domain to the frequency domain;
• obtain compensated frequency spectrum information based on the frequency spectrum information of the target audio data through a generator of the GAN model, the generator including a plurality of compensation subunits, each one of the plurality of compensation subunits including an upsampling layer and a convolution layer;
• generate predicted audio data corresponding to the time interval in the time domain based on the compensated frequency spectrum information;
• obtain compensated audio data based on modification of a portion of the target audio data according to a corresponding portion of the predicted audio data, the portion of the target audio data including the audio missing segment and a subset of the context audio segments;
• determine difference information between the compensated audio data and the training audio data through a discriminator of the GAN model;
• and
• update the generator or the generator and the discriminator based on the difference information.