# US 2025/0165122 A1 · Providing composite graphical assistant interfaces for controlling various connected devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0165122 A1
- Document ID: US20250165122A1
- Status: Published application
- Kind: A1
- Application number: 19/029,520
- Filed: 2025-01-17
- Published: 2025-05-22
- Earliest claimed: 2018-05-07
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Yuzhao Ni; David Roy Schairer
- Web page: https://patentgrep.com/patent/US20250165122A1

## CPC

- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- H04L 67/125: involving control of end-device applications over a network
- G06F 3/04812: Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- H04L 12/282: {based on user interaction within the home (receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204)}
- H04L 67/12: specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
- H04L 67/306: User profiles
- H04L 67/51: Discovery or management thereof, e.g. service location protocol [SLP] or web services
- H04L 67/535: {Tracking the activity of the user (network monitoring arrangements H04L43/00; recording of computer activity G06F11/34)}
- G10L 2015/223: {Execution procedure of a spoken command}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Methods, apparatus, systems, and computer-readable media are provided for tailoring composite graphical assistant interfaces for interacting with multiple different connected devices. The composite graphical assistant interfaces can be generated in response to a user providing a request for an automated assistant to cause a connected device to perform a particular function. In response to the automated assistant receiving the request, the automated assistant can identify other connected devices, and other functions capable of being performed by the other connected devices. The other functions can then be mapped to various graphical control elements in order to provide a composite graphical assistant interface from which the user can interact with different connected devices. Each graphical control element can be arranged to reflect how each connected device is operating simultaneous to the presentation of the composite graphical assistant interface.

## Claims

### Claim 1

A method implemented by one or more processors, the method comprising:
determining, based on processing historical interaction data, a historical inclination of one or more users to request an automated assistant to cause performance of a second function during a particular temporal context;
determining that a spoken utterance, detected via an automated assistant interface of a client device, includes a request for the automated assistant to cause a first connected device, that is linked to the client device, to perform the first function;
determining that a current temporal context of the spoken utterance corresponds to the particular temporal context;
identifying, in response to determining that the current temporal context of the spoken utterance corresponds to the particular temporal context, a second connected device that is linked to the client device, wherein identifying the second connected device is based on the historical inclination of one or more of the users to cause performance the second function in the particular temporal context and the second function being performable via the second connected device;
determining a graphical control element for causing the second connected device to perform the second function; and
in response to determining the spoken utterance includes the request for the automated assistant to cause the first connected device to perform the first function:
causing the first connected device to perform the first function; and
causing a display panel associated with the client device to provide the graphical control element for causing the second connected device to perform the second function.

### Claim 2 (depends on 1)

The method of claim 1, wherein the one or more users include a particular user that provided the spoken utterance.

### Claim 3 (depends on 1)

The method of claim 1, wherein the one or more users include one or more additional users that are different from a particular user that provided the spoken utterance.

### Claim 4 (depends on 1)

The method of claim 1, wherein determining the graphical control element for causing the second connected device to perform the second function is based on a current status of the second connected device.

### Claim 5 (depends on 4)

The method of claim 4, wherein determining the graphical control element for causing the second connected device to perform the second function comprises:
determining the current status of the second connected device;
determining a previous status of the second connected device after a previous instance of the second connected device performing the second function;
comparing the current status of the second connected device to the previous status of the second connected device; and
determining the graphical control element for causing the second connected device to perform the second function based on the current status being different from the previous status.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
identifying a third connected device that is linked to the client device, wherein identifying the third connected device is based on a historical inclination of one or more of the users to change a status of the third connected device while one or more of the users are in the particular temporal context; and
determining a second graphical control element for changing the status of the third connected device, wherein the display panel further includes the second graphical control element.

### Claim 7 (depends on 6)

The method of claim 6, wherein the graphical control element and the second graphical element are arranged according to a determined configuration for each graphical control element.

### Claim 8

A system comprising:
memory storing instructions; and
one or more processors operable to execute the instructions to:
determine, based on processing historical interaction data, a historical inclination of one or more users to request an automated assistant to cause performance of a second function during a particular temporal context;
determine that a spoken utterance, detected via an automated assistant interface of a client device, includes a request for the automated assistant to cause a first connected device, that is linked to the client device, to perform the first function;
determine that a current temporal context of the spoken utterance corresponds to the particular temporal context;
identify, in response to determining that the current temporal context of the spoken utterance corresponds to the particular temporal context, a second connected device that is linked to the client device, wherein identifying the second connected device is based on the historical inclination of one or more of the users to cause performance the second function in the particular temporal context and the second function being performable via the second connected device;
determine a graphical control element for causing the second connected device to perform the second function; and
in response to determining the spoken utterance includes the request for the automated assistant to cause the first connected device to perform the first function:
cause the first connected device to perform the first function; and
cause a display panel associated with the client device to provide the graphical control element for causing the second connected device to perform the second function.

### Claim 9 (depends on 8)

The system of claim 8, wherein the one or more users include a particular user that provided the spoken utterance.

### Claim 10 (depends on 8)

The system of claim 8, wherein the one or more users include one or more additional users that are different from a particular user that provided the spoken utterance.

### Claim 11 (depends on 8)

The system of claim 8, wherein determining the graphical control element for causing the second connected device to perform the second function is based on a current status of the second connected device.

### Claim 12 (depends on 11)

The system of claim 11, wherein in determining the graphical control element for causing the second connected device to perform the second function, one or more of the processors are to
determine the current status of the second connected device;
determine a previous status of the second connected device after a previous instance of the second connected device performing the second function;
compare the current status of the second connected device to the previous status of the second connected device; and
determine the graphical control element for causing the second connected device to perform the second function based on the current status being different from the previous status.

### Claim 13 (depends on 8)

The system of claim 8, wherein one or more of the processors are further to:
identify a third connected device that is linked to the client device, wherein identifying the third connected device is based on a historical inclination of one or more of the users to change a status of the third connected device while one or more of the users are in the particular temporal context; and
determine a second graphical control element for changing the status of the third connected device, wherein the display panel further includes the second graphical control element.

### Claim 14 (depends on 13)

The system of claim 13, wherein the graphical control element and the second graphical element are arranged according to a determined configuration for each graphical control element.

### Claim 15

A non-transitory computer readable storage medium configured to store instructions that, when executed by one or more processors, cause one or more of the processors to:
determine, based on processing historical interaction data, a historical inclination of one or more users to request an automated assistant to cause performance of a second function during a particular temporal context;
determine that a spoken utterance, detected via an automated assistant interface of a client device, includes a request for the automated assistant to cause a first connected device, that is linked to the client device, to perform the first function;
determine that a current temporal context of the spoken utterance corresponds to the particular temporal context;
identify, in response to determining that the current temporal context of the spoken utterance corresponds to the particular temporal context, a second connected device that is linked to the client device, wherein identifying the second connected device is based on the historical inclination of one or more of the users to cause performance the second function in the particular temporal context and the second function being performable via the second connected device;
determine a graphical control element for causing the second connected device to perform the second function; and
in response to determining the spoken utterance includes the request for the automated assistant to cause the first connected device to perform the first function:
cause the first connected device to perform the first function; and
cause a display panel associated with the client device to provide the graphical control element for causing the second connected device to perform the second function.

### Claim 16 (depends on 15)

The non-transitory computer readable storage medium of claim 15, wherein the one or more users include a particular user that provided the spoken utterance.

### Claim 17 (depends on 15)

The non-transitory computer readable storage medium of claim 15, wherein the one or more users include one or more additional users that are different from a particular user that provided the spoken utterance.

### Claim 18 (depends on 15)

The non-transitory computer readable storage medium of claim 15, wherein determining the graphical control element for causing the second connected device to perform the second function is based on a current status of the second connected device.

### Claim 19 (depends on 18)

The non-transitory computer readable storage medium of claim 18, wherein in determining the graphical control element for causing the second connected device to perform the second function, one or more of the processors are to
determine the current status of the second connected device;
determine a previous status of the second connected device after a previous instance of the second connected device performing the second function;
compare the current status of the second connected device to the previous status of the second connected device; and
determine the graphical control element for causing the second connected device to perform the second function based on the current status being different from the previous status.

### Claim 20 (depends on 15)

The non-transitory computer readable storage medium of claim 15, wherein one or more of the processors are further to:
identify a third connected device that is linked to the client device, wherein identifying the third connected device is based on a historical inclination of one or more of the users to change a status of the third connected device while one or more of the users are in the particular temporal context; and
determine a second graphical control element for changing the status of the third connected device, wherein the display panel further includes the second graphical control element.
