System

A system comprising:

• a communication interface including a network interface controller coupled to a packet-switched network and configured to transmit and receive data packets;
• a memory storing instructions, a deep learning speech recognition model comprising a plurality of self-attention layers and feed-forward layers, a neural network speech synthesis model comprising a neural acoustic model and a neural vocoder, and a generative neural network model;
• and
• circuitry comprising one or more processors coupled to the memory and configured to execute the instructions to:
• receive, via the communication interface, audio data representing an utterance of a user captured by an acoustic sensor of a terminal device coupled to the packet-switched network;
• perform digital signal processing on the audio data including noise suppression in a frequency domain, frequency band emphasis, and dynamic range correction to generate a processed speech signal;
• convert the processed speech signal into time-series feature data including mel-frequency feature vectors normalized using per-user statistics, and input the time-series feature data to the deep learning speech recognition model to generate a character string representing content of the utterance;
• correct the character string by performing phrase replacement based on utterance history information and frequent-expression information associated with the user;
• generate prosody control information for the corrected character string and input the prosody control information and voice output attributes associated with the user to the neural network speech synthesis model to generate a synthesized speech signal;
• transmit the synthesized speech signal to the terminal device via the communication interface for acoustic output;
• and
• construct a prompt data structure including meta-information relating to at least speech characteristics of the user, an acoustic environment of the terminal device, and a recognition error history, transmit the prompt data structure to the generative neural network model, receive response data, and update a configuration parameter of at least one of the deep learning speech recognition model and the neural network speech synthesis model based on the response data.

A method performed by circuitry of a server coupled to a packet-switched network via a communication interface, the method comprising:

• receiving, via the communication interface, audio data representing an utterance of a user captured by an acoustic sensor of a terminal device;
• performing digital signal processing on the audio data to generate a processed speech signal in which features of the utterance are emphasized;
• converting the processed speech signal into time-series feature data and performing speech recognition using a deep learning model based on the time-series feature data to generate a character string representing content of the utterance;
• generating a synthesized speech signal based on the character string using a speech synthesis model;
• transmitting the synthesized speech signal to the terminal device via the communication interface for acoustic output;
• and
• transmitting a prompt data structure to a generative neural network model and updating a parameter of at least one of the deep learning model and the speech synthesis model based on response data received from the generative neural network model.