# US 12,183,346 B2 · Hub device, multi-device system including the hub device and plurality of devices, and method of operating the same > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,183,346 B2 - Document ID: US12183346B2 - Status: Granted - Kind: B2 - Application number: 18/336,652 - Filed: 2023-06-16 - Published: 2023-10-19 - Granted: 2024-12-31 - Earliest claimed: 2019-05-02 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Yeonho Lee; Sangwook Park; Kookjin Yeo - Web page: https://patentgrep.com/patent/US12183346B2 ## CPC - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L 15/18: using natural language modelling - G10L 15/1822: {Parsing for meaning understanding} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 2015/223: {Execution procedure of a spoken command} - G10L 2015/228: {of application context} - G16Y 20/20: relating to the thing itself - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G16Y: INFORMATION AND COMMUNICATION TECHNOLOGY SPECIALLY ADAPTED FOR THE INTERNET OF THINGS [IoT] ## Abstract A hub device, a multi-device system including the hub device, and a method of operating the same may include: converting, by the hub device, received voice input into text; identifying, by the hub device, a device capable of performing an operation corresponding to the text; identifying which device stores a function determination model corresponding to the device capable of performing the operation corresponding to the text, from among the hub device, and a plurality of other devices connected to the hub device; and based on the identified device that stores the function determination model being a device that is different from the hub device, transmitting at least part of the text to the identified device. ## Claims ### Claim 1 A hub device for controlling a device based on a voice input, the hub device comprising: a communication interface configured to perform data communication with at least one of a plurality of devices, a voice assistant server, or an Internet of things (IoT) server; a microphone configured to receive a voice input of a user; a memory storing one or more instructions; and at least one processor configured to execute the one or more instructions stored in the memory to: convert the voice input received by the microphone into text by performing automatic speech recognition (ASR), determine an operation-performing device from among the plurality of devices based on the text, by using a device determination model, identify a device that stores a function determination model corresponding to the determined operation-performing device, and control the communication interface to provide at least part of the text to the identified device. ### Claim 2 (depends on 1) The hub device of claim 1, wherein the device determination model comprises a first natural language understanding (NLU) model configured to analyze the text and determine the operation-performing device based on a result of the analyzing of the text. ### Claim 3 (depends on 1) The hub device of claim 1, wherein the function determination model comprises a second NLU model configured to analyze at least part of the text and obtain operation information about an operation to be performed by the determined operation-performing device based on a result of the analyzing of the at least part of the text. ### Claim 4 (depends on 1) The hub device of claim 1, wherein the at least one processor is further configured to determine whether the function determination model corresponding to the determined operation-performing device is stored in the memory. ### Claim 5 (depends on 4) The hub device of claim 4, wherein the function determination model corresponding to the determined operation-performing device is stored in the memory of the hub device, and the at least one processor is further configured to control the communication interface to provide the at least part of the text to the function determination model stored in the memory. ### Claim 6 (depends on 5) The hub device of claim 5, wherein the at least one processor is further configured to: determine intent by interpreting the at least part of the text by using a second NLU model of the function determination model stored in the memory, obtain operation information about operation to be performed by the determined operation-performing device based on the determined intent, generate control command based on the operation information, and control the communication interface to transmit the control command to the determined operation-performing device. ### Claim 7 (depends on 1) The hub device of claim 1, wherein the at least one processor is further configured to: determine, based on information about the function determination model, whether the function determination model corresponding to the determined operation-performing device is stored in a memory of the operation-performing device, and based on a result of the determination, control the communication interface to transmit the at least part of the text to the function determination model stored in the memory of the operation-performing device. ### Claim 8 (depends on 1) The hub device of claim 1, wherein the function determination model corresponding to the determined operation-performing device is stored in the voice assistant server, the at least one processor is further configured to control the communication interface to transmit the at least part of the text to the function determination model stored in the voice assistant server. ### Claim 9 (depends on 1) The hub device of claim 1, wherein the plurality of devices are logged in by the same user account as an user account of the hub device and connected to the hub device through a network. ### Claim 10 A method, performed by a hub device, of controlling a device based on a voice input, the method comprising: receiving a voice input of a user; converting the received voice input into text by performing automatic speech recognition (ASR); determining an operation-performing device based on the text, by using a device determination model; identifying a device that stores a function determination model corresponding to the determined operation-performing device; and providing at least part of the text to the identified device. ### Claim 11 (depends on 10) The method of claim 10, wherein the device determination model comprises a first natural language understanding (NLU) model configured to analyze the text and determine the operation-performing device based on a result of the analyzing of the text. ### Claim 12 (depends on 10) The method of claim 10, wherein the function determination model comprises a second NLU model configured to analyze at least part of the text and obtain operation information about an operation to be performed by the determined operation-performing device based on a result of the analyzing of the at least part of the text. ### Claim 13 (depends on 10) The method of claim 10, wherein the identifying of the device that stores the function determination model comprises determining whether the function determination model corresponding to the determined operation-performing device is stored in the memory. ### Claim 14 (depends on 13) The method of claim 13, wherein the function determination model corresponding to the determined operation-performing device is stored in the memory of the hub device, and the providing of the at least part of the text comprises providing the at least part of the text to the function determination model stored in the memory. ### Claim 15 (depends on 14) The method of claim 14, further comprising: determining intent by interpreting the at least part of the text by using a second NLU model of the function determination model stored in the memory; obtaining operation information about operation to be performed by the determined operation-performing device based on the determined intent; generating control command based on the operation information; and transmitting the control command to the determined operation-performing device. ### Claim 16 (depends on 10) The method of claim 10, wherein the identifying of the device that stores the function determination model comprises determining, based on information about the function determination model, whether the function determination model corresponding to the determined operation-performing device is stored in a memory of the operation-performing device, and the providing of the at least part of the text comprises transmitting the at least part of the text to the function determination model stored in the memory of the operation-performing device based on a result of the determination. ### Claim 17 (depends on 10) The method of claim 10, wherein the function determination model corresponding to the determined operation-performing device is stored in the voice assistant server, the providing of the at least part of the text comprises transmitting the at least part of the text to the function determination model stored in the voice assistant server. ### Claim 18 (depends on 10) The method of claim 10, wherein a plurality of devices are logged in by the same user account as an user account of the hub device and connected to the hub device through a network. ### Claim 19 A non-transitory computer-readable recording medium having recorded thereon a program for executing a method comprising: receiving a voice input of a user; converting the received voice input into text by performing automatic speech recognition (ASR); determining an operation-performing device based on the text, by using a device determination model; identifying a device that stores a function determination model corresponding to the determined operation-performing device; and providing at least part of the text to the identified device.