Methods, systems, and apparatus for providing composite graphical assistant interfaces for controlling connected devices

A system comprising:

• one or more processors;
• and
• memory configured to store instructions that, when executed by the one or more processors cause the one or more processors to perform operations that include:
• determining that a spoken utterance, detected at a client device and via an automated assistant interface of an automated assistant, includes a request to modify a state of a connected device that is linked with the client device, wherein the spoken utterance is provided by a user of the automated assistant;
• identifying, using the automated assistant and in response to receiving the request, a first function that corresponds to the request and first slot values being control parameters for controlling the first function;
• causing, based on identifying the function, the connected device to perform the first function that corresponds to the request;
• determining, using the automated assistant, a current state of the connected device, wherein the current state of the connected device is affected by performance of the first function;
• identifying, using the automated assistant, one or more other functions, capable of being performed by the connected device, and other slot values for the one or more other functions, wherein the current state of the connected device is further affected by performance of the one or more other functions;
• determining, based on detecting the spoken utterance, to render a composite graphical assistant interface, wherein the composite graphical assistant interface was not rendered when the spoken utterance was detected;
• in response to determining to render the composite graphical assistant interface:
• determining, using the automated assistant and in response to identifying the first function and the one or more other functions, a first graphical control element based on a number and/or type of the first slot values for selecting a first slot value to control the first function performed by the connected device and a second graphical control element based on a number and/or type of the other slot values for selecting one or more of the other slot values to control the one or more other functions;
• determining, using the automated assistant and based on the current state of the connected device, a configuration for each graphical control element of the first graphical control element and the second graphical control element;
• and
• causing, in response to receiving the request, the automated assistant to render the composite graphical assistant interface for controlling the connected device via a display panel associated with the client device,
• wherein the composite graphical assistant interface includes the first graphical control element and the second graphical control element, which are arranged according to the determined configuration for each graphical control element.

A non-transitory computer readable storage medium configured to store instructions that, when executed by one or more processors, cause the one or more processors to perform operations that include:

• determining that a spoken utterance, detected at a client device and via an automated assistant interface of an automated assistant, includes a request to modify a state of a connected device that is linked with the client device, wherein the spoken utterance is provided by a user of the automated assistant;
• identifying, using the automated assistant and in response to receiving the request, a first function that corresponds to the request and first slot values being control parameters for controlling the first function;
• causing, based on identifying the function, the connected device to perform the first function that corresponds to the request;
• determining, using the automated assistant, a current state of the connected device, wherein the current state of the connected device is affected by performance of the first function;
• identifying, using the automated assistant, one or more other functions, capable of being performed by the connected device, and other slot values for the one or more other functions, wherein the current state of the connected device is further affected by performance of the one or more other functions;
• determining, based on detecting the spoken utterance, to render a composite graphical assistant interface, wherein the composite graphical assistant interface was not rendered when the spoken utterance was detected;
• in response to determining to render the composite graphical assistant interface:
• determining, using the automated assistant and in response to identifying the first function and the one or more other functions, a first graphical control element based on a number and/or type of the first slot values for selecting a first slot value to control the first function performed by the connected device and a second graphical control element based on a number and/or type of the other slot values for selecting one or more of the other slot values to control the one or more other functions;
• determining, using the automated assistant and based on the current state of the connected device, a configuration for each graphical control element of the first graphical control element and the second graphical control element;
• and
• causing, in response to receiving the request, the automated assistant to render the composite graphical assistant interface for controlling the connected device via a display panel associated with the client device,
• wherein the composite graphical assistant interface includes the first graphical control element and the second graphical control element, which are arranged according to the determined configuration for each graphical control element.