# US 10,872,438 B2 · Artificial intelligence device capable of being controlled according to user's gaze and method of operating the same > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,872,438 B2 - Document ID: US10872438B2 - Status: Granted - Kind: B2 - Application number: 16/543,152 - Filed: 2019-08-16 - Published: 2019-12-05 - Granted: 2020-12-22 - Earliest claimed: 2019-07-01 - Assignees: LG Electronics Inc. - Applicants: LG Electronics Inc. - Inventors: Taeju Hwang - Web page: https://patentgrep.com/patent/US10872438B2 ## CPC - G06T 7/75: {involving models} - G05B 13/027: {using neural networks only} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06N 3/008: based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour - G06N 3/0499: Feedforward networks - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G05B 2219/32335: Use of ann, neural network - G05B 2219/33027: Artificial neural network controller - G06N 3/04: Architecture, e.g. interconnection topology - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30201: Face - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract An artificial intelligence (AI) device capable of being controlled according to a user's gaze includes a communication unit, a camera configured to capture an image of a user, and a processor configured to acquire user state information from the image of the user, acquire a gaze position of the user based on the acquired user state information, calculate a distance between the acquired gaze position and the camera, receive, from one or more external AI devices, one or more distances between gaze positions of the user respectively acquired by the external AI devices and cameras respectively provided in the external AI devices through the communication unit, and compare the calculated distance with the received one or more distances to select a controlled device. ## Claims ### Claim 1 An artificial intelligence (AI) device comprising: a communication circuit; a camera; and one or more processors configured to: acquire user state information from one or more images of a user captured via the camera; acquire a gaze position of the user based on the acquired user state information; calculate a first distance between the acquired gaze position and the camera; receive, from at least one external AI device via the communication circuit, a respective calculated distance between the gaze position of the user and a corresponding camera of each of the at least one external AI device; and select a device as a controlled device based on a comparison of the first calculated distance and the respective calculated distances of each of the at least one external AI device. ### Claim 2 (depends on 1) The AI device of claim 1, wherein a device having a minimum distance among the first calculated distance and the respective calculated distances is selected as the controlled device. ### Claim 3 (depends on 2) The AI device of claim 2, wherein the one or more processors are further configured to transmit to the at least one external AI device a signal indicating that the AI device is selected as the controlled device and that an operation state of the at least one external AI device is to be changed to an inactive state based on the AI device having the minimum distance. ### Claim 4 (depends on 2) The AI device of claim 2, wherein the one or more processors are further configured to transmit to one the at least one external AI device a signal indicating that the one external AI device is selected as the controlled device based on the one external AI device having the minimum distance. ### Claim 5 (depends on 1) The AI device of claim 1, wherein the user state information includes a face direction, a face angle, and a gaze direction of the user. ### Claim 6 (depends on 5) The AI device of claim 5, further comprising a memory, wherein: the one or more processors are further configured to acquire the gaze position using the user state information as an input to a gaze position recognition model stored in the memory; and the gaze position recognition model is an artificial neural network. ### Claim 7 (depends on 6) The AI device of claim 6, wherein: training data of the gaze position recognition model includes the user state information and gaze coordinates as labeling data; and the gaze position recognition model is trained such that a difference between an actual position of the gaze and gaze coordinates determined by the gaze position recognition model is minimized. ### Claim 8 (depends on 6) The AI device of claim 6, wherein the gaze position is indicated by three-dimensional gaze coordinates, and the one or more processors are further configured to calculate a distance between coordinates corresponding to the camera and the gaze coordinates, wherein the coordinates corresponding to the camera are represented as an origin. ### Claim 9 (depends on 3) The AI device of claim 3, further comprising a microphone, wherein the one or more processors are further configured to acquire an intention of a voice command of a user received via the microphone and cause an operation to be performed based on the acquired intention. ### Claim 10 (depends on 1) The AI device of claim 1, wherein the one or more processors are further configured to calculate the first distance between the acquired gaze position and the camera after recognizing an eyeblink of the user based on the one or more images of the user captured via the camera. ### Claim 11 A method of operating an artificial intelligence (AI) device, the method comprising: capturing one or more images of a user; acquiring user state information from the one or more images of the user; acquiring a gaze position of the user based on the acquired user state information; calculating a first distance between the acquired gaze position and the camera; receiving, from at least one external AI device, a respective calculated distance between the gaze position of the user and a corresponding camera of each of the at least one external AI device; and selecting a device as a controlled device based on a comparison of the first calculated distance and the respective calculated distances of each of the at least one external AI device. ### Claim 12 (depends on 11) The method of claim 11, wherein a device having a minimum distance among the first calculated distance and the respective calculated distances is selected as the controlled device. ### Claim 13 (depends on 12) The method of claim 12, further comprising transmitting to the at least one external AI device a signal indicating that the AI device is selected as the controlled device and that an operation state of the at least one external AI device is to be changed to an inactive state based on the AI device having the minimum distance. ### Claim 14 (depends on 12) The method of claim 12, further comprising transmitting to one of the at least one external AI device a signal indicating that the one external AI device is selected as the controlled device based on the one external AI device having the minimum distance. ### Claim 15 (depends on 11) The method of claim 11, wherein: the user state information includes a face direction, a face angle, and a gaze direction of the user; the gaze position is acquired using the user state information and a gaze position recognition model; and the gaze position recognition model is an artificial neural network.