Assistive control of network-connected devices

A non-transitory computer-readable medium storing instructions which, when executed by a processor located in a communication network, wherein the processor is remote from a home network that is connected to the communication network via a cellular access network, cause the processor to perform operations, the operations comprising:

• obtaining, from a computing device located in the home network via the cellular access network, images of a face of a user;
• detecting a first facial gesture of the user from the images;
• obtaining, from the computing device via the cellular access network, a first non-speech utterance of the user;
• determining an intention to change a state of a network-connected device in the home network, wherein the network-connected device is separate from the computing device, and wherein the intention is determined from the first facial gesture and the first non-speech utterance based upon a mapping of the user, and wherein the mapping of the user is generated via a machine learning process that learns that the first facial gesture in combination with the first non-speech utterance signifies the intention to change the state of the network-connected device;
• generating an audio command to cause the state of the network-connected device to change in accordance with the intention, when the computing device confirms, in connection with a response of the user that includes a second facial gesture and a second non-speech utterance, the intention to change the state of the network-connected device;
• and
• providing, via the cellular access network, the audio command to the computing device along with an instruction for the computing device to play the audio command via a speaker of the computing device for detection via a short range wireless communication by an interactive voice assistant device, separate from the computing device and located in the home network, to change the state of the network-connected device by the interactive voice assistant device.

A method comprising:

• obtaining, by a processing system located in a communication network, wherein the processing system includes at least one processor and is remote from a home network that is connected to the communication network via a cellular access network, images of a face of a user in the home network, wherein the images are received via the cellular access network from a computing device located in the home network;
• detecting, by the processing system, a first facial gesture of the user from the images;
• obtaining, by the processing system via the cellular access network, a first non-speech utterance of the user;
• determining, by the processing system, an intention to change a state of a network-connected device in the home network, wherein the network-connected device is separate from the computing device, and wherein the intention is determined from the first facial gesture and the first non-speech utterance based upon a mapping of the user, and wherein the mapping of the user is generated via a machine learning process that learns that the first facial gesture in combination with the first non-speech utterance signifies the intention to change the state of the network-connected device;
• generating, by the processing system, an audio command to cause the state of the network-connected device to change in accordance with the intention, when the computing device confirms, in connection with a response of the user that includes a second facial gesture and a second non-speech utterance, the intention to change the state of the network-connected device;
• and
• providing, by the processing system via the cellular access network, the audio command to the computing device along with an instruction for the computing device to play the audio command via a speaker of the computing device for detection via a short range wireless communication by an interactive voice assistant device, separate from the computing device and located in the home network, to change the state of the network-connected device by the interactive voice assistant device.