# US 12,112,769 B2 · System, user terminal, and method for providing automatic interpretation service based on speaker separation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,769 B2 - Document ID: US12112769B2 - Status: Granted - Kind: B2 - Application number: 17/531,316 - Filed: 2021-11-19 - Published: 2022-07-07 - Granted: 2024-10-08 - Earliest claimed: 2021-01-05 - Assignees: Electronics and Telecommunications Research Institute - Applicants: Electronics and Telecommunications Research Institute - Inventors: Jeong Uk Bang; Seung Yun; Sang Hun Kim; Min Kyu Lee; Joon Gyu Maeng - Web page: https://patentgrep.com/patent/US12112769B2 ## CPC - G10L 25/84: for discriminating voice from noise - G06F 40/30: Semantic analysis - G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) - G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation - G10L 13/00: Speech synthesis; Text to speech systems - G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit - G10L 15/08: Speech classification or search - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L 21/0208: Noise filtering - G10L 21/0272: Voice signal separating - H04M 2203/2061: Language aspects - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08) ## Abstract Provided is a method of performing automatic interpretation based on speaker separation by a user terminal, the method including: receiving a first speech signal including at least one of a user speech of a user and a user surrounding speech around the user from an automatic interpretation service providing terminal, separating the first speech signal into speaker-specific speech signals, performing interpretation on the speaker-specific speech signals in a language selected by the user on the basis of an interpretation mode, and providing a second speech signal generated as a result of the interpretation to at least one of a counterpart terminal and the automatic interpretation service providing terminal according to the interpretation mode. ## Claims ### Claim 1 A method of performing automatic interpretation based on speaker separation by a user terminal, the method comprising: receiving a first speech signal including at least one of a user speech of a user and a user surrounding speech around the user from an automatic interpretation service providing terminal; separating the first speech signal into speaker-specific speech signals; performing interpretation on the speaker-specific speech signals in a language selected by the user on the basis of an interpretation mode; and providing a second speech signal generated as a result of the interpretation to at least one of a counterpart terminal and the automatic interpretation service providing terminal according to the interpretation mode, wherein the performing of interpretation on the speaker-specific speech signals in the language selected by the user on the basis of the interpretation mode includes: extracting situation information from the user surrounding speech; wherein the providing of the second speech signal generated as the result of the interpretation to the at least one of the counterpart terminal and the automatic interpretation service providing terminal according to the interpretation mode includes: classifying speech signals according to each speaker from the extracted situation information; and providing the automatic interpretation service providing terminal with an interpretation result in which intensity information and echo information of the speech signals classified according to each speaker are reflected. ### Claim 2 (depends on 1) The method of claim 1, wherein the receiving of the first speech signal including the at least one of the user speech of the user and the user surrounding speech includes receiving the first speech signal on the basis of the interpretation mode from the automatic interpretation service providing terminal, wherein the automatic interpretation service providing terminal receives the user speech through a microphone when the interpretation mode is a conversation mode and receives the user surrounding speech through the microphone when the interpretation mode is a listening mode. ### Claim 3 (depends on 1) The method of claim 1, wherein the performing of interpretation on the speaker-specific speech signals in the language selected by the user on the basis of the interpretation mode includes, when the interpretation mode is a conversation mode, selecting the user speech among the separated speaker-specific speech signals to perform the interpretation, and the providing of the second speech signal generated as the result of the interpretation to the at least one of the counterpart terminal and the automatic interpretation service providing terminal according to the interpretation mode includes providing the counterpart terminal with the second speech signal, which is an interpretation result for the user speech, and providing the automatic interpretation service providing terminal with an interpretation result for a counterpart speech received from the counterpart terminal. ### Claim 4 (depends on 1) The method of claim 1, wherein the performing of interpretation on the speaker-specific speech signals in the language selected by the user on the basis of the interpretation mode includes, when the interpretation mode is a listening mode, selecting the user surrounding speech among the separated speaker-specific speech signals to perform the interpretation, and the providing of the second speech signal generated as the result of the interpretation to the at least one of the counterpart terminal and the automatic interpretation service providing terminal according to the interpretation mode includes classifying interpretation results for the user surrounding speech according to each speaker and providing the automatic interpretation service providing terminal with the interpretation results. ### Claim 5 (depends on 4) The method of claim 4, wherein the performing of interpretation on the speaker-specific speech signals into the language selected by the user on the basis of the interpretation mode includes: extracting the situation information including at least one of speaker information, context information, and noise information from the user surrounding speech; storing the extracted situation information in a buffer; and interpreting the speaker-specific speech signal included in the user surrounding speech on the basis of the extracted situation information. ### Claim 6 (depends on 4) The method of claim 4, wherein the providing of the second speech signal generated as the result of the interpretation to the at least one of the counterpart terminal and the automatic interpretation service providing terminal according to the interpretation mode includes providing the automatic interpretation service providing terminal with an interpretation result corresponding to at least one speaker selected by the user among the interpretation results for the user surrounding speech. ### Claim 7 A user terminal for providing an automatic interpretation service based on speaker separation, the user terminal comprising: a communication module configured to transmit and receive data to and from an automatic interpretation service providing terminal and a counterpart terminal; a memory in which a program for receiving a first speech signal including at least one of a user speech of a user and a user surrounding speech around the user from the automatic interpretation service providing terminal and, based on the received first speech signal, generating a second speech signal that is an interpretation result is stored; and a processor configured to execute the program stored in the memory, wherein the processor executes the program to: separate the first speech signal into speaker-specific speech signals; perform interpretation on the speaker-specific speech signals in a language selected by the user on the basis of an interpretation mode; and provide a second speech signal generated as a result of the interpretation to at least one of a counterpart terminal and the automatic interpretation service providing terminal through the communication module according to the interpretation mode, wherein the processor, when the interpretation mode is a listening mode, selects the user surrounding speech among the separated speaker-specific speech signals to perform the interpretation, classifies interpretation results for the user surrounding speech according to each speaker, and provides the automatic interpretation service providing terminal with the interpretation results through the communication module, wherein the processor extracts situation information including at least one of speaker information, context information, and noise information from the user surrounding speech, stores the extracted situation information in a buffer, and interprets the speaker-specific speech signal included in the user surrounding speech on the basis of the extracted situation information, and wherein the processor is configured to, based on the first and second speech signals and the extracted situation information, automatically update performances of a speaker-specific source separation model for separating the first speech signal into the speaker-specific speech signals, a speech recognition model for recognizing the first speech signal, a machine translation model for translating the first speech signal, and an automatic interpretation model having a combination thereof. ### Claim 8 (depends on 7) The user terminal of claim 7, wherein the automatic interpretation service providing terminal receives the user speech through a microphone when the interpretation mode is a conversation mode and receives the user surrounding speech through the microphone when the interpretation mode is a listening mode. ### Claim 9 (depends on 7) The user terminal of claim 7, wherein the processor, when the interpretation mode is a conversation mode, selects the user speech among the separated speaker-specific speech signals to perform the interpretation, provides the counterpart terminal with the second speech signal, which is an interpretation result for the user speech, through the communication module, and provides the automatic interpretation service providing terminal with an interpretation result for a counterpart speech received from the counterpart terminal. ### Claim 10 (depends on 7) The user terminal of claim 7, wherein the processor classifies speech signals according to each speaker from the extracted situation information and provides the automatic interpretation service providing terminal with an interpretation result, in which intensity information and echo information of the speech signal classified according to each speaker are reflected, through the communication module. ### Claim 11 (depends on 7) The user terminal of claim 7, wherein the processor provides the automatic interpretation service providing terminal with an interpretation result corresponding to at least one speaker selected by the user among the interpretation results for the user surrounding speech. ### Claim 12 A system for providing an automatic interpretation service based on speaker separation, the system comprising: an automatic interpretation service providing terminal configured to receive a first speech signal including at least one of a user speech of a user and a user surrounding speech around the user through a multichannel microphone and provide an interpretation result for the speech signal and an interpretation result for a counterpart speech through a speaker; a user terminal configured to receive the first speech signal, separate the received first speech signal into speaker-specific signals, perform interpretation on the speaker-specific speech signals in a language selected by the user on the basis of an interpretation mode, and provide a second speech signal generated as a result of the interpretation to at least one of a counterpart terminal and the automatic interpretation service providing terminal according to the interpretation mode; and the counterpart terminal configured to receive the second speech signal from the user terminal and provide the received second speech signal, interpret the counterpart speech and provide the interpreted counterpart speech according to the interpretation mode, wherein the user terminal is configured to extract situation information from the user surrounding speech, provide of the second speech signal generated as the result of the interpretation to the at least one of the counterpart terminal and the automatic interpretation service providing terminal according to the interpretation mode, classify speech signals according to each speaker from the extracted situation information, and provide the automatic interpretation service providing terminal with an interpretation result in which intensity information and echo information of the speech signals classified according to each speaker are reflected.