System, user terminal, and method for providing automatic interpretation service based on speaker separation

A user terminal for providing an automatic interpretation service based on speaker separation, the user terminal comprising:

• a communication module configured to transmit and receive data to and from an automatic interpretation service providing terminal and a counterpart terminal;
• a memory in which a program for receiving a first speech signal including at least one of a user speech of a user and a user surrounding speech around the user from the automatic interpretation service providing terminal and, based on the received first speech signal, generating a second speech signal that is an interpretation result is stored;
• and
• a processor configured to execute the program stored in the memory,
• wherein the processor executes the program to:
• separate the first speech signal into speaker-specific speech signals;
• perform interpretation on the speaker-specific speech signals in a language selected by the user on the basis of an interpretation mode;
• and
• provide a second speech signal generated as a result of the interpretation to at least one of a counterpart terminal and the automatic interpretation service providing terminal through the communication module according to the interpretation mode,
• wherein the processor, when the interpretation mode is a listening mode, selects the user surrounding speech among the separated speaker-specific speech signals to perform the interpretation, classifies interpretation results for the user surrounding speech according to each speaker, and provides the automatic interpretation service providing terminal with the interpretation results through the communication module,
• wherein the processor extracts situation information including at least one of speaker information, context information, and noise information from the user surrounding speech, stores the extracted situation information in a buffer, and interprets the speaker-specific speech signal included in the user surrounding speech on the basis of the extracted situation information, and
• wherein the processor is configured to, based on the first and second speech signals and the extracted situation information, automatically update performances of a speaker-specific source separation model for separating the first speech signal into the speaker-specific speech signals, a speech recognition model for recognizing the first speech signal, a machine translation model for translating the first speech signal, and an automatic interpretation model having a combination thereof.

A system for providing an automatic interpretation service based on speaker separation, the system comprising:

• an automatic interpretation service providing terminal configured to receive a first speech signal including at least one of a user speech of a user and a user surrounding speech around the user through a multichannel microphone and provide an interpretation result for the speech signal and an interpretation result for a counterpart speech through a speaker;
• a user terminal configured to receive the first speech signal, separate the received first speech signal into speaker-specific signals, perform interpretation on the speaker-specific speech signals in a language selected by the user on the basis of an interpretation mode, and provide a second speech signal generated as a result of the interpretation to at least one of a counterpart terminal and the automatic interpretation service providing terminal according to the interpretation mode;
• and
• the counterpart terminal configured to receive the second speech signal from the user terminal and provide the received second speech signal, interpret the counterpart speech and provide the interpreted counterpart speech according to the interpretation mode,
• wherein the user terminal is configured to extract situation information from the user surrounding speech, provide of the second speech signal generated as the result of the interpretation to the at least one of the counterpart terminal and the automatic interpretation service providing terminal according to the interpretation mode, classify speech signals according to each speaker from the extracted situation information, and provide the automatic interpretation service providing terminal with an interpretation result in which intensity information and echo information of the speech signals classified according to each speaker are reflected.