# US 2024/0203398 A1 · Multi-modal voice recognition system and method for conversation summarization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0203398 A1
- Document ID: US20240203398A1
- Status: Published application
- Kind: A1
- Application number: 18/540,594
- Filed: 2023-12-14
- Published: 2024-06-20
- Earliest claimed: 2022-12-19
- Assignees: Electronics and Telecommunications Research Institute
- Applicants: Electronics and Telecommunications Research Institute
- Inventors: Jeom Ja KANG; Kiyoung PARK; Hwajeon SONG
- Web page: https://patentgrep.com/patent/US20240203398A1

## CPC

- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G06N 3/02: Neural networks
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Disclosed herein is a voice recognition system with an enhanced summarization function according to the present invention. The voice recognition system include: an audio feature extractor configured to extract a voice feature from an audio signal to generate a feature vector; a salience extractor configured to extract a importance of speech from at least one of the audio signal or a video signal to generate an importance vector; and a neural network configured to output a recognition result based on the feature vector and the importance vector, in which the recognition result is output by masking some.

## Claims

### Claim 1

A voice recognition system comprising:
an audio feature extractor configured to extract a voice feature from an audio signal to generate a feature vector;
an audio/video salience extractor configured to extract an importance of speech from at least one of the audio signal or video signal to generate an importance vector; and
a neural network configured to output a recognition result based on the feature vector and the importance vector, wherein the recognition result is output by masking some words.

### Claim 2 (depends on 1)

The voice recognition system of claim 1, wherein the voice recognition system is a multi-modal system configured to acquire both voice and video.

### Claim 3 (depends on 1)

The voice recognition system of claim 1, wherein the audio/video salience extractor extracts magnitude and pitch information on the voice signal from the audio signal to extract a word or segment that is emphasized or of high interest by a speaker.

### Claim 4 (depends on 1)

The voice recognition system of claim 1, wherein the audio/video salience extractor extracts, in addition to the audio signal, information that is indicative of interest in speech content, including gaze information on participants, from the video signal.

### Claim 5 (depends on 1)

The voice recognition system of claim 1, further comprising:
a preprocessor configured to concatenate the feature vector and the importance vector for each unit interval.

### Claim 6 (depends on 1)

The voice recognition system of claim 1, wherein the neural network includes:
an acoustic model generator configured to generate an acoustic model based on the feature vector and the importance vector; and
a recognition result output unit configured to output a recognition result based on the acoustic model, wherein the recognition result is output by masking some words.

### Claim 7 (depends on 6)

The voice recognition system of claim 6, wherein the recognition result output unit masks a word with the degree of importance below a predetermined reference in the target output.

### Claim 8 (depends on 6)

The voice recognition system of claim 6, wherein the recognition result output unit selects the words to be masked in a manner that masks a predetermined number of words in low rank with low importance depending on an output of the audio/video salience extractor.

### Claim 9 (depends on 1)

The voice recognition system of claim 1, wherein the audio/video salience extractor performs training for judging importance based on the masked words.

### Claim 10

A multi-modal voice recognition system comprising:
an audio feature extractor configured to extract a voice feature from an audio signal to generate a feature vector;
a salience extractor configured to extract an importance of speech from the audio signal and the video signal to generate an importance vector;
a preprocessing unit configured to concatenate the feature vector and the importance vector by a predetermined interval;
an encoder configured to generate an acoustic model based on the concatenated vector;
a buffer configured to store an output of the encoder; and
a decoder configured to output a recognition result based on the output of the encoder, wherein the recognition result is output by masking some words in the target output.

### Claim 11

A voice recognition method comprising:
generating a voice feature vector by extracting a voice feature from an audio signal;
generating an importance vector by measuring an importance of speech;
generating an acoustic model based on the feature vector and the importance vector;
outputting a recognition result based on the acoustic model, wherein the recognition result is output by masking words having the degree of importance below a predetermined reference; and
retraining a voice recognition neural network based on the masked output.

### Claim 12 (depends on 11)

The voice recognition method of claim 11, further comprising:
concatenating to correlate the feature vector and the importance vector by a predetermined identical interval.

### Claim 13 (depends on 11)

The voice recognition method of claim 11, wherein the generating of the importance vector comprises:
extracting magnitude and pitch information on the voice signal from the audio signal; and
extracting, from the video signal, gesture information in units of frames that is indicative of interest in speech content, including gaze information of participants.

### Claim 14 (depends on 11)

The voice recognition method of claim 11, wherein the masking and outputting of the recognition result comprises masking all except a predetermined number of words in high rank with high importance in the output.
