System

A system comprising:

• circuitry configured to:
• receive, via a communication interface coupled to a packet-switched network, time-series image data and audio data from a terminal device;
• apply an image processing program to the image data to detect a face region and a body region of a subject and extract facial feature information including facial expression state, gaze state, and head posture state, and posture feature information including body posture state and apparel state;
• apply a speech recognition program to the audio data to generate audio text data and apply an acoustic analysis program to the audio data to extract acoustic feature information including tone state, fluency state, and pronunciation state, and apply a natural language processing program to the audio text data to extract semantic feature information;
• normalize the facial feature information, the posture feature information, the acoustic feature information, and the semantic feature information and concatenate the normalized information into a unified feature representation following a predetermined schema;
• input the unified feature representation into a machine learning model to generate machine learning scores for evaluation elements including facial expression, body posture, apparel, tone, fluency, pronunciation, and answer content;
• generate a prompt sentence including the machine learning scores, selected feature summaries, and an evaluation instruction, and input the prompt sentence and the unified feature representation into a generative neural network model to obtain element scores and evaluation comment information;
• apply a weighting algorithm using weighting parameters to the machine learning scores and the element scores to compute a comprehensive evaluation score, operate an emotion engine on the unified feature representation to generate an emotion score, and dynamically correct the weighting parameters based on the emotion score;
• and
• store the comprehensive evaluation score, the element scores, the evaluation comment information, and the emotion score in a storage device indexed by an evaluation session identifier, and transmit the comprehensive evaluation score and the evaluation comment information to the terminal device via the communication interface.

A method comprising:

• receiving, via a communication interface coupled to a packet-switched network, time-series audiovisual data from a terminal device, the time-series audiovisual data including image data and audio data;
• applying an image processing program to the image data to extract, for each of a plurality of visual evaluation elements, a feature vector representing facial expression state, gaze state, head posture state, body posture state, and apparel state of a subject;
• applying a speech recognition program and an acoustic analysis program to the audio data to extract audio text data and acoustic feature information, and deriving, from the audio text data and the acoustic feature information, feature vectors representing tone state, fluency state, and pronunciation state of the subject;
• applying a natural language processing program to the audio text data to extract semantic feature information representing at least structure, relevance, and concreteness of answer content of the subject;
• normalizing and concatenating the feature vectors and the semantic feature information into a unified feature representation according to a predetermined data structure;
• inputting the unified feature representation into a machine learning model trained on a past evaluation data set to generate machine learning scores for respective evaluation elements;
• generating a prompt sentence incorporating the machine learning scores and selected feature summaries, and inputting the prompt sentence and the unified feature representation into a generative neural network model to obtain element scores and evaluation comment information from the generative neural network model;
• integrating the machine learning scores and the element scores into integrated evaluation scores for the respective evaluation elements, applying a weighting algorithm using weighting parameters to compute a comprehensive evaluation score, and operating an emotion engine on the unified feature representation to generate an emotion score;
• and
• storing the comprehensive evaluation score, the integrated evaluation scores, the evaluation comment information, and the emotion score in a storage device, and transmitting the comprehensive evaluation score and the evaluation comment information to the terminal device via the communication interface.