Multi-modal voice recognition system and method for conversation summarization

A multi-modal voice recognition system comprising:

• an audio feature extractor configured to extract a voice feature from an audio signal to generate a feature vector;
• a salience extractor configured to extract an importance of speech from the audio signal and the video signal to generate an importance vector;
• a preprocessing unit configured to concatenate the feature vector and the importance vector by a predetermined interval;
• an encoder configured to generate an acoustic model based on the concatenated vector;
• a buffer configured to store an output of the encoder;
• and
• a decoder configured to output a recognition result based on the output of the encoder, wherein the recognition result is output by masking some words in the target output.

A voice recognition method comprising:

• generating a voice feature vector by extracting a voice feature from an audio signal;
• generating an importance vector by measuring an importance of speech;
• generating an acoustic model based on the feature vector and the importance vector;
• outputting a recognition result based on the acoustic model, wherein the recognition result is output by masking words having the degree of importance below a predetermined reference;
• and
• retraining a voice recognition neural network based on the masked output.