# US 12,009,941 B2 · Assistive control of network-connected devices > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,009,941 B2 - Document ID: US12009941B2 - Status: Granted - Kind: B2 - Application number: 18/161,876 - Filed: 2023-01-30 - Published: 2023-06-01 - Granted: 2024-06-11 - Earliest claimed: 2018-12-14 - Assignees: AT&T Intellect al P Property I, L.P. - Applicants: AT&T Intellectual Property I, L.P. - Inventors: Forest Johnson; Pamela Juhl Sokoler; Prakash Thiruvenkatam - Web page: https://patentgrep.com/patent/US12009941B2 ## CPC - H04L 12/2816: {Controlling appliance services of a home automation network by calling their functionalities (arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station; in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom H04Q9/00)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06V 40/176: {Dynamic expression} - H04L 41/0813: characterised by the conditions triggering a change of settings - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Devices, computer-readable media, and methods for changing the state of a network-connected device in response to at least one facial gesture of a user are disclosed. For example, a processing system including at least one processor captures images of a face of a user, detects at least one facial gesture of the user from the images, determines an intention to change a state of a network-connected device from the at least one facial gesture, generates a command for the network-connected device in accordance with the intention, and outputs the command to cause the state of the network-connected device to change. ## Claims ### Claim 1 A device comprising: a processing system in a communication network including at least one processor, wherein the processing system is remote from a home network that is connected to the communication network via a cellular access network; and a non-transitory computer-readable medium storing instructions which, when executed by the processing system, cause the processing system to perform operations, the operations comprising: obtaining, from a computing device located in the home network via the cellular access network, images of a face of a user; detecting a first facial gesture of the user from the images; obtaining, from the computing device via the cellular access network, a first non-speech utterance of the user; determining an intention to change a state of a network-connected device in the home network, wherein the network-connected device is separate from the computing device, and wherein the intention is determined from the first facial gesture and the first non-speech utterance based upon a mapping of the user, and wherein the mapping of the user is generated via a machine learning process that learns that the first facial gesture in combination with the first non-speech utterance signifies the intention to change the state of the network-connected device; generating an audio command to cause the state of the network-connected device to change in accordance with the intention, when the computing device confirms, in connection with a response of the user that includes a second facial gesture and a second non-speech utterance, the intention to change the state of the network-connected device; and providing, via the cellular access network, the audio command to the computing device along with an instruction for the computing device to play the audio command via a speaker of the computing device for detection via a short range wireless communication by an interactive voice assistant device, separate from the computing device and located in the home network, to change the state of the network-connected device by the interactive voice assistant device. ### Claim 2 (depends on 1) The device of claim 1, wherein the computing device includes a camera. ### Claim 3 (depends on 1) The device of claim 1, wherein the detecting the first facial gesture of the user from the images is in accordance with the machine learning model for the first facial gesture. ### Claim 4 (depends on 1) The device of claim 1, wherein the processing system maintains a plurality of user-specific mappings for a plurality of different users, and the mapping of the user is one of the plurality of user-specific mappings. ### Claim 5 (depends on 4) The device of claim 4, wherein each mapping of the plurality of user-specific mappings is created by at least one of: a user with whom the each mapping is associated; or a person assisting the user with whom the each mapping is associated. ### Claim 6 (depends on 1) The device of claim 1, wherein the first facial gesture is part of a sequence of facial gestures. ### Claim 7 (depends on 1) The device of claim 1, wherein the first facial gesture comprises at least one of: a blink; a wink; a head nod; a head shake; or a frown. ### Claim 8 (depends on 1) The device of claim 1, wherein the intention is determined from a sequence of the first facial gesture and the first non-speech utterance. ### Claim 9 (depends on 1) The device of claim 1, wherein the operations further comprise: detecting a presence of the user in the home network, wherein the detecting the first facial gesture of the user is activated in response to the detecting the presence of the user. ### Claim 10 (depends on 9) The device of claim 9, where the presence of the user is detected via the images of the face of the user. ### Claim 11 (depends on 9) The device of claim 9, wherein the presence of the user is detected via: a detection of a wireless device of the user. ### Claim 12 (depends on 9) The device of claim 9, where the presence of the user is detected via a radio frequency identification tag of the user. ### Claim 13 (depends on 1) The device of claim 1, wherein the operations further comprise: activating the network-connected device by detecting a third facial gesture prior to performing the obtaining the images, the detecting the first facial gesture, the obtaining the first non-speech utterance, the determining the intention, the generating the audio command, and the providing the audio command. ### Claim 14 (depends on 13) The device of claim 13, wherein the third facial gesture is mapped in the mapping to a wake up word that activates the network-connected device. ### Claim 15 (depends on 1) The device of claim 1, wherein the mapping is stored in a database that is accessible by the processing system. ### Claim 16 (depends on 1) The device of claim 1, wherein the network-connected device is owned by someone other than the user. ### Claim 17 (depends on 16) The device of claim 16, wherein the someone other than the user subscribes to a network-based service provided by the device. ### Claim 18 A non-transitory computer-readable medium storing instructions which, when executed by a processor located in a communication network, wherein the processor is remote from a home network that is connected to the communication network via a cellular access network, cause the processor to perform operations, the operations comprising: obtaining, from a computing device located in the home network via the cellular access network, images of a face of a user; detecting a first facial gesture of the user from the images; obtaining, from the computing device via the cellular access network, a first non-speech utterance of the user; determining an intention to change a state of a network-connected device in the home network, wherein the network-connected device is separate from the computing device, and wherein the intention is determined from the first facial gesture and the first non-speech utterance based upon a mapping of the user, and wherein the mapping of the user is generated via a machine learning process that learns that the first facial gesture in combination with the first non-speech utterance signifies the intention to change the state of the network-connected device; generating an audio command to cause the state of the network-connected device to change in accordance with the intention, when the computing device confirms, in connection with a response of the user that includes a second facial gesture and a second non-speech utterance, the intention to change the state of the network-connected device; and providing, via the cellular access network, the audio command to the computing device along with an instruction for the computing device to play the audio command via a speaker of the computing device for detection via a short range wireless communication by an interactive voice assistant device, separate from the computing device and located in the home network, to change the state of the network-connected device by the interactive voice assistant device. ### Claim 19 A method comprising: obtaining, by a processing system located in a communication network, wherein the processing system includes at least one processor and is remote from a home network that is connected to the communication network via a cellular access network, images of a face of a user in the home network, wherein the images are received via the cellular access network from a computing device located in the home network; detecting, by the processing system, a first facial gesture of the user from the images; obtaining, by the processing system via the cellular access network, a first non-speech utterance of the user; determining, by the processing system, an intention to change a state of a network-connected device in the home network, wherein the network-connected device is separate from the computing device, and wherein the intention is determined from the first facial gesture and the first non-speech utterance based upon a mapping of the user, and wherein the mapping of the user is generated via a machine learning process that learns that the first facial gesture in combination with the first non-speech utterance signifies the intention to change the state of the network-connected device; generating, by the processing system, an audio command to cause the state of the network-connected device to change in accordance with the intention, when the computing device confirms, in connection with a response of the user that includes a second facial gesture and a second non-speech utterance, the intention to change the state of the network-connected device; and providing, by the processing system via the cellular access network, the audio command to the computing device along with an instruction for the computing device to play the audio command via a speaker of the computing device for detection via a short range wireless communication by an interactive voice assistant device, separate from the computing device and located in the home network, to change the state of the network-connected device by the interactive voice assistant device. ### Claim 20 (depends on 19) The method of claim 19, wherein the computing device comprises a wearable device.