System

A system comprising:

• circuitry configured to:
• receive, via a communication interface coupled to a packet-switched network, signal data representing an utterance captured by a terminal device;
• extract an acoustic feature sequence from the signal data and input the acoustic feature sequence to a neural network model comprising a sequence-to-sequence architecture with an acoustic encoder and a linguistic decoder to generate a token sequence, and convert the token sequence into character data;
• normalize user attribute data received from the terminal device, the user attribute data indicating at least one of a perceptual characteristic and a cognitive characteristic, and select a prompt template from a plurality of stored prompt templates on the basis of the user attribute data and a processing request type;
• construct a prompt data structure from the selected prompt template, generate generation input data including the prompt data structure and the character data, and input the generation input data to a generative neural network model comprising a transformer architecture with a token embedding layer, a positional encoding mechanism, a plurality of self-attention layers, and feedforward sublayers to generate output character data;
• estimate an emotional state of a user associated with the terminal device by applying an emotion classification model to analysis target data derived from at least one of the signal data and image data received from the terminal device, the emotion classification model fusing at least two of a pitch contour, an energy envelope, a speaking rate, and facial action unit values to generate a probabilistic representation of the emotional state;
• when the emotional state indicates at least one of confusion and anxiety, construct a rewrite prompt data structure and input the rewrite prompt data structure together with the output character data to the generative neural network model to generate adjusted character data having at least one of reduced linguistic difficulty and increased repetition;
• insert prosody control markers into the adjusted character data or the output character data, the prosody control markers specifying at least one of a speech rate, a pitch value, a pause duration, and an emphasis indicator determined on the basis of the emotional state and the user attribute data;
• input the adjusted character data or the output character data with the prosody control markers to a synthesis model comprising an encoder-decoder with an attention mechanism and a vocoder to generate output signal data;
• and
• transmit the output signal data to the terminal device via the packet-switched network.

A method performed by circuitry of a server coupled to a packet-switched network via a communication interface, the method comprising:

• receiving, via the communication interface, input data including at least one of character data and signal data from a terminal device;
• converting the signal data into character data by applying a neural network model that maps an acoustic feature sequence to a token sequence;
• generating a prompt data structure designating at least one of a compression operation and a transformation operation on the character data, and constructing generation input data including the prompt data structure and the character data;
• inputting the generation input data to a generative neural network model to generate output character data representing a condensed or restructured form of the character data;
• converting the output character data into output signal data by applying a synthesis model that generates a waveform representation;
• and
• transmitting the output signal data to the terminal device via the packet-switched network.