Systems and methods for generating context-oriented summary of audio conversations in real time

A method for generating context-oriented summary of audio conversations in real time, the method comprising:

• pre-training a principal model with training audio data to generate summary, and the principal model includes a transformer-based model and a hybrid summary generation module, wherein the transformer-based model comprises a spectrogram generation module, a One Dimensional Convolutional Neural Network (1D CNN) layer, a sinusoidal positional embedding module, an encoder and a decoder;
• receiving by the pre-trained principal model, real-time audio data which is processed by the transformer-based model to generate a series of encoder hidden state tensors, output by the encoder of the transformer-based model, wherein the encoder includes convolutional neural networks (CNN) layers followed by self-attention layers;
• transforming by an independent self-attention layer, the series of encoder hidden state tensors into a new set of hidden states based on additional weighted attention scores of the independent self-attention layer;
• transforming by a mean pooling module, the new set of hidden states into a single fixed-length tensor representation;
• and
• selecting by the decoder, relevant features from the single fixed-length tensor representation by utilizing the additional weighted attention scores of the independent self-attention layer and generate context-oriented summaries of the real-time audio data.