# US 12,217,016 B2 · Electronic apparatus for translating voice input using neural network models and method for controlling thereof > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,016 B2 - Document ID: US12217016B2 - Status: Granted - Kind: B2 - Application number: 17/746,238 - Filed: 2022-05-17 - Published: 2022-11-17 - Granted: 2025-02-04 - Earliest claimed: 2021-05-17 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Beomseok Lee; Sathish Indurthi; Mohd Abbas Zaidi; Nikhil Kumar - Web page: https://patentgrep.com/patent/US12217016B2 ## CPC - G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G06F 40/44: Statistical methods, e.g. probability models - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/08: Learning methods - G10L 13/02: Methods for producing synthetic speech; Speech synthesisers - G10L 15/16: using artificial neural networks - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract An electronic apparatus, including a microphone; a memory configured to store at least one instruction; and a processor configured to: acquire a first token corresponding to a first user voice input in a first language acquired through the microphone, acquire a first text in a second language by inputting the first token into a first neural network model, acquire a feature value corresponding to a predicted subsequent token, which is predicted to be uttered after the first token, by inputting the first text into a second neural network model, and based on a second token being acquired subsequent to the first token, acquire a second text in the second language by inputting the first token, the second token, the first text, and the feature value into the first neural network model. ## Claims ### Claim 1 An electronic apparatus comprising: a microphone; a memory configured to store at least one instruction; and a processor configured to: acquire a first token corresponding to a first user voice input in a first language acquired through the microphone, acquire a first text in a second language by inputting the first token into a first neural network model, acquire a feature value corresponding to a predicted subsequent token, which is predicted to be uttered after the first token, by inputting the first text into a second neural network model, and based on a second token being acquired subsequent to the first token, acquire a second text in the second language by inputting the first token, the second token, the first text, and the feature value into the first neural network model, wherein the first neural network model acquires the second text based on a first feature related to a grammatical relationship between the first token and the second token, a second feature related to a semantic relationship between the first token and the second token and the feature value corresponding to the predicted subsequent token. ### Claim 2 (depends on 1) The electronic apparatus of claim 1, wherein based on an input token of the first language being inputted into the first neural network model, the first neural network model is trained to acquire text of the second language corresponding to the input token, or to identify an additional input token in addition to the input token. ### Claim 3 (depends on 2) The electronic apparatus of claim 2, wherein the first neural network model further comprises a first encoder and a first decoder, and wherein, based on a context vector being acquired by inputting the input token into the first encoder, the first decoder is trained to acquire the text of the second language corresponding to the input token. ### Claim 4 (depends on 3) The electronic apparatus of claim 3, wherein the first decoder is configured to, based on a probability value acquired based on the context vector being greater than a predetermined value, acquire the text of the second language corresponding to the input token, and based on the probability value acquired based on the context vector being less than the predetermined value, identify an additional token in addition to the input token. ### Claim 5 (depends on 1) The electronic apparatus of claim 1, further comprising a display, wherein the processor is further configured to control the display to display the first text and the second text. ### Claim 6 (depends on 1) The electronic apparatus of claim 1, further comprising: a speaker, wherein the processor is further configured to control the speaker to output a voice message corresponding to the first text and the second text. ### Claim 7 A method for controlling an electronic apparatus comprising: acquiring a first token corresponding to a first user voice input in a first language; acquiring a first text in a second language by inputting the first token into a first neural network model; acquiring a feature value corresponding to a predicted subsequent token, which is predicted to be uttered after the first token, by inputting the first text into a second neural network model; and based on a second token being acquired subsequent to the first token, acquiring a second text in the second language by inputting the first token, the second token, the first text, and the feature value into the first neural network model, wherein the first neural network model acquires the second text based on a first feature related to a grammatical relationship between the first token and the second token, a second feature related to a semantic relationship between the first token and the second token and the feature value corresponding to the predicted subsequent token. ### Claim 8 (depends on 7) The method of claim 7, wherein, based on an input token of the first language being inputted into the first neural network model, the first neural network model to acquire text of the second language corresponding to the input token or to identify an additional input token in addition to the input token. ### Claim 9 (depends on 8) The method of claim 8, wherein the first neural network model further comprises a first encoder and a first decoder, and wherein, based on a context vector being acquired by inputting the input token into the first encoder, the first decoder is configured to acquire the text of the second language corresponding to the input token. ### Claim 10 (depends on 9) The method of claim 9, wherein the first decoder is configured to, based on a probability value acquired based on the context vector being greater than a predetermined value, acquire the text of the second language corresponding to the input token, and based on the probability value acquired based on the context vector being less than the predetermined value, identify an additional token in addition to the input token. ### Claim 11 (depends on 7) The method of claim 7, further comprising displaying the first text and the second text. ### Claim 12 (depends on 7) The method of claim 7, further comprising outputting a voice message corresponding to the first text and the second text. ### Claim 13 A non-transitory computer-readable recording medium storing a program which, when executed by at least one processor, causes the at least one processor to: acquire a first token corresponding to a first user voice input in a first language; acquire a first text in a second language by inputting the first token into a first neural network model; acquire a feature value corresponding to a predicted subsequent token predicted, which is predicted to be uttered after the first token, by inputting the first text into a second neural network model; and based on a second token being acquired subsequent to the first token, acquire a second text in the second language by inputting the first token, the second token, the first text, and the feature value into the first neural network model, wherein the first neural network model acquires the second text based on a first feature related to a grammatical relationship between the first token and the second token, a second feature related to a semantic relationship between the first token and the second token and the feature value corresponding to the predicted subsequent token.