# US 12,008,988 B2 · Electronic apparatus and controlling method thereof > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,988 B2 - Document ID: US12008988B2 - Status: Granted - Kind: B2 - Application number: 17/065,027 - Filed: 2020-10-07 - Published: 2021-04-15 - Granted: 2024-06-11 - Earliest claimed: 2019-10-10 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Hyeontaek Lim; Sejin Kwak; Youngjin Kim - Web page: https://patentgrep.com/patent/US12008988B2 ## CPC - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 40/35: Discourse or dialogue representation - G10L 15/18: using natural language modelling - G10L 15/24: Speech recognition using non-acoustical features - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract An electronic apparatus and a controlling method thereof are provided. The electronic apparatus includes a microphone, a camera, a memory configured to store at least one command, and at least one processor configured to, based on a first user voice being input from a user, provide a response to the first user voice, based on an audio signal including a voice being input while the response to the first user voice is provided, analyze an image captured by the camera and determine whether there is a second user voice uttered by the user in the audio signal, and based on determining that there is the second user voice uttered by the user in the audio signal, stop providing the response to the first user voice and obtain and provide a response to the second user voice. ## Claims ### Claim 1 An electronic apparatus comprising: a microphone; a camera; a memory configured to store at least one command; and at least one processor configured to be connected to the microphone, the camera, and the memory and control the electronic apparatus, wherein the at least one processor is further configured, by executing the at least one command, to: receive a first voice input from a first user through the microphone, using a first natural language understanding model at the electronic apparatus to identify a first category information of the first voice input and provide a response to the first voice input, receive a second voice input through the microphone while the response to the first voice input is being provided, analyze an image captured by the camera and determine whether the second voice input is uttered by the first user based on the captured image, in response to determining that the second voice input is uttered by the first user, stop the providing the response to the first voice input and using the first natural language understanding model to identify second category information in the second voice input, determine whether the first category in the first voice input is being related to the second category in the second voice input, in response to the first category information in the first voice input is being related to the second category information in the second voice input, use the first natural language understanding model to perform natural language understanding on the second voice input, and in response to the first category information in the first voice input is being unrelated to the second category information in the second voice input, control a communication interface to transmit the second voice input to a server and using a second natural language understanding model at the server to perform natural language understanding on the second voice input. ### Claim 2 (depends on 1) The electronic apparatus of claim 1, wherein the at least one processor is further configured to: based on a third voice input including a wake-up word being input by the first user, recognize the wake-up word and register the first user included in an image captured by the camera; extract an area including the registered user in the image captured by the camera while the response to the first voice input is being provided; and determine whether the second voice input uttered by the registered user is in an audio signal by analyzing the extracted area including the registered user. ### Claim 3 (depends on 2) The electronic apparatus of claim 2, wherein the at least one processor is further configured to: extract a lip area of the registered user in the image captured by the camera; and determine whether the second voice input uttered by the registered user is in the audio signal by analyzing whether there is a movement of the extracted lip area of the registered user. ### Claim 4 (depends on 1) The electronic apparatus of claim 1, wherein the at least one processor is further configured to, based on an audio signal being input while the response to the first voice input is being provided, store the audio signal in a buffer without regard to whether the second voice input uttered by the first user is in the audio signal. ### Claim 5 (depends on 4) The electronic apparatus of claim 4, wherein the at least one processor is further configured to: based on determining that the second voice input uttered by the first user is in the audio signal, obtain and provide the response to the second voice input based on the audio signal stored in the buffer; and based on determining that the second voice input uttered by the first user is not in the audio signal, ignore the audio signal stored in the buffer. ### Claim 6 (depends on 1) The electronic apparatus of claim 1, wherein the at least one processor is further configured to: based on determining that the second voice input uttered by the first user is in an audio signal, identify a sentence type of the second voice input; and perform natural language understanding regarding the second voice input, based on the identified sentence type. ### Claim 7 (depends on 6) The electronic apparatus of claim 6, wherein the at least one processor is further configured to, based on the identified sentence type being interrogative, control to transmit information regarding the second voice input to a server providing the second natural language understanding model. ### Claim 8 (depends on 1) The electronic apparatus of claim 1, wherein the at least one processor is further configured to: based on the first voice input being input, store the first voice input and information regarding the response to the first voice input in the memory as conversation history information of the first user; and based on determining that the second voice input uttered by the first user is in an audio signal, obtain and provide the response to the second voice input based on the second voice input and the conversation history information. ### Claim 9 (depends on 1) The electronic apparatus of claim 1, wherein the at least one processor is further configured to: determine a priority of the second voice input; and based on a determination that the priority of the second voice input is higher than a threshold value, control to transmit information regarding the second voice input to a server providing the second natural language understanding model. ### Claim 10 A controlling method of an electronic apparatus, the method comprising: receiving a first voice input from a first user through a microphone, using a first natural language understanding model at the electronic apparatus to identify a first category information of the first voice input and provide a response to the first voice input; receiving a second voice input through the microphone while the response to the first voice input is being provided, analyze an image captured by a camera and determine whether the second voice input is uttered by the first user based on the captured image; in response to determining that the second voice input is uttered by the first user, stopping the providing the response to the first voice input and using the first natural language understanding model to identify second category information in the second voice input; determining whether the first category in the first voice input is being related to the second category in the second voice input; in response to the first category information in the first voice input is being related to the second category information in the second voice input, using the first natural language understanding model to perform natural language understanding on the second voice input; and in response to the first category information in the first voice input is being unrelated to the second category information in the second voice input, transmitting the second voice input to a server and using a second natural language understanding model at the server to perform natural language understanding on the second voice input. ### Claim 11 (depends on 10) The method of claim 10, further comprising: based on a third voice input including a wake-up word being input by the first user, recognizing the wake-up word and registering the first user included in an image captured by the camera, wherein the determining that the second voice input uttered by the first user is in an audio signal comprises: extracting an area including the registered user in the image captured by the camera while the response to the first voice input is being provided; and determining whether the second voice input uttered by the registered user is in the audio signal by analyzing the extracted area including the registered user. ### Claim 12 (depends on 11) The method of claim 11, wherein the determining that the second voice input uttered by the first user is in the audio signal further comprises: extracting a lip area of the registered user in the image captured by the camera; and determining whether the second voice input uttered by the registered user is in the audio signal by analyzing whether there is a movement of the extracted lip area of the registered user. ### Claim 13 (depends on 10) The method of claim 10, further comprising: based on an audio signal being input while the response to the first voice input is being provided, storing the audio signal in a buffer without regard to whether the second voice input uttered by the first user is in the audio signal. ### Claim 14 (depends on 13) The method of claim 13, wherein the providing of the response to the second voice input comprises: based on determining that the second voice input uttered by the first user is in the audio signal, obtaining a response to the second voice input based on the audio signal stored in the buffer, and based on determining that the second voice input uttered by the first user is not in the audio signal, ignoring the audio signal stored in the buffer. ### Claim 15 (depends on 10) The method of claim 10, wherein the providing of the response to the second voice input comprises: based on determining that the second voice input uttered by the first user is in an audio signal, identifying a sentence type of the second voice input; and performing natural language understanding regarding the second voice input based on the identified sentence type. ### Claim 16 (depends on 10) The method of claim 10, further comprising: based on the first voice input being input, storing the first voice input and information regarding the response to the first voice input in a memory as conversation history information of the first user, based on determining that the second voice input uttered by the first user is in an audio signal, obtaining and providing the response to the second voice input based on the second voice input and the conversation history information.