Method and apparatus for neural spatial speech coding for multi-channel audio

A codec for performing neural spatial audio coding, comprising:

• at least one memory configured to store program code;
• and
• at least one processor configured to read the program code and operate as instructed by the program code, the program code including:
• receiving code configured to cause the at least one processor to receive an audio signal comprising a plurality of channels;
• selecting code configured to cause the at least one processor to select a channel from the plurality of channels as a reference channel;
• first performing code configured to cause the at least one processor to perform a short time fourier transform (STFT) on the reference channel to generate a frequency domain reference channel;
• first inputting code configured to cause the at least one processor to input the frequency domain reference channel into a first codec that comprises a first neural network for encoding the frequency domain reference channel and a second neural network for decoding the frequency domain reference channel;
• second performing code configured to cause the at least one processor to perform the STFT on the plurality of channels minus the channel selected as the reference channel to generate a spatial covariance matrix;
• second inputting code configured to cause the at least one processor to input the spatial covariance matrix and the frequency domain reference channel into a second codec that comprises a third neural network for encoding the spatial covariance matrix and the frequency domain reference channel and a fourth neural network for decoding the spatial covariance matrix;
• reconstructing code configured to cause the at least one processor to reconstruct the audio signal based on an output of the first codec and an output of the second codec to generate a reconstructed audio signal;
• first training code configured to cause the at least one processor to train the first codec based on a comparison of the output of the first codec with the frequency domain reference channel;
• and
• second training code configured to cause the at least one processor to train the second codec based on a comparison of the reconstructed audio signal with the plurality of channels of the audio signal minus the reference channel.

A non-transitory computer readable medium having instructions stored therein, which when executed by a processor in a codec for performing neural spatial audio coding, cause the processor to execute a method comprising:

• receiving an audio signal comprising a plurality of channels;
• selecting a channel from the plurality of channels as a reference channel;
• performing a short time fourier transform (STFT) on the reference channel to generate a frequency domain reference channel;
• inputting the frequency domain reference channel into a first codec that comprises a first neural network for encoding the frequency domain reference channel and a second neural network for decoding the frequency domain reference channel;
• performing the STFT on the plurality of channels minus the channel selected as the reference channel to generate a spatial covariance matrix;
• inputting the spatial covariance matrix and the frequency domain reference channel into a second codec that comprises a third neural network for encoding the spatial covariance matrix and the frequency domain reference channel and a fourth neural network for decoding the spatial covariance matrix;
• reconstructing the audio signal based on an output of the first codec and an output of the second codec to generate a reconstructed audio signal;
• training the first codec based on a comparison of the output of the first codec with the frequency domain reference channel;
• and
• training the second codec based on a comparison of the reconstructed audio signal with the plurality of channels of the audio signal minus the reference channel.