US 12,223,954 B2Granted

Automated assistant control of external applications lacking automated assistant application programming interface functionality

AssigneeGoogle Inc.

InventorsThomas Deselaers, Sandro Feuz

Life of this patent

Estimating the term…

Loading patent…

Abstract

Implementations relate to an automated assistant that is capable of interacting with non-assistant applications that do not have functionality explicitly provided for interfacing with certain automated assistants. Application data, such as annotation data and/or GUI data, associated with a non-assistant application, can be processed to map such data into an embedding space. An assistant input command can then be processed and mapped to the same embedding space, and a distance from the assistant input command embedding and the non-assistant application data embedding can be determined. When the distance between the assistant input command embedding and the non-assistant application data embedding satisfies threshold(s), the automated assistant can generate instruction(s), for the non-assistant application, that correspond to the non-assistant application data. For instance, the instruction(s) can simulate user input(s) that cause the non-assistant application to perform one or more operations characterized by, or otherwise associated with, the non-assistant application data.

Claims15

Dependent claims

A method implemented by one or more processors, the method further comprising:

causing, at a computing device, a suggestion graphical user interface (GUI) element to be rendered at a display interface of the computing device,
wherein the suggestion GUI element includes natural language content embodying a suggested command for a user to provide to an automated assistant that is accessible via the computing device,
wherein the suggested command corresponds to a request for the automated assistant to cause a particular application to perform one or more operations, and
wherein the particular application is void of a function that allows the particular application to be controlled by the automated assistant without the user initializing the particular application prior to the automated assistant controlling the particular application;
wherein causing the suggestion GUI element to be rendered at the display interface of the computing device includes:
causing the suggestion GUI element to be rendered in a foreground of the display interface of the computing device when the particular application is also being rendered at the display interface of the computing device;
and
when the user provides a spoken utterance that includes the suggested command:
generating an instance of training data that indicates the user provided the suggested command as an input to the automated assistant,
wherein the instance of training data is generated for training a machine learning model that can be used for processing a subsequent spoken utterance from the user.

7 more elements

A method implemented by one or more processors, the method comprising:

initializing, at a computing device, a particular application that is accessible via the computing device,
wherein the computing device also provides access to an automated assistant, and
wherein the particular application is void of a function that allows one or more operations of the particular application to be controlled by the automated assistant;
and
causing, in response to initializing the particular application, a graphical user interface (GUI) to be rendered at a display interface of the computing device,
wherein the particular application includes application data that characterizes the one or more operations, which are capable of being initialized in response to an input from the user to the GUI of the particular application;
receiving, from the automated assistant, a request for the particular application to perform the one or more operations,
wherein the request is provided by the automated assistant in response to the automated assistant receiving an input from the user;
and
wherein receiving the request from the automated assistant includes determining that the request identifies a particular GUI element that is being rendered at the display interface of the computing device;
and
causing, in response to receiving the request from the automated assistant, the particular application to perform the one or more operations.

9 more elements

A method implemented by one or more processors, the method comprising:

initializing, at a computing device, a particular application that is accessible via the computing device,
wherein the computing device also provides access to an automated assistant, and
wherein the particular application is void of a function that allows one or more operations of the particular application to be controlled by the automated assistant;
and
causing, in response to initializing the particular application, a graphical user interface (GUI) to be rendered at a display interface of the computing device,
wherein the particular application includes application data that characterizes the one or more operations, which are capable of being initialized in response to an input from the user to the GUI of the particular application;
receiving, from the automated assistant, a request for the particular application to perform the one or more operations,
wherein the request is provided by the automated assistant in response to the automated assistant receiving an input from the user;
and
wherein receiving the request from the automated assistant includes determining that the request includes input data for simulating user interaction with a particular GUI element that is being rendered at the display interface of the computing device;
and
causing, in response to receiving the request from the automated assistant, the particular application to perform the one or more operations.

9 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/058,895Filed 10 Nov 2020 · us provisional application
Priority: 2 Jun 2020
Published: US 2024/0249719 A125 Jul 2024
Granted: 11 Feb 2025
Term ends: Not estimatedThe source records don’t support an estimate.

Classification10 codes

G10L 15/22
Speech recognition: procedures used during a speech recognition process, e.g. man-machine dialogue
G06F 3/167
Sound input; Sound output: audio in a user interface, e.g. using voice commands for navigating, audio feedback
G06F 9/547
Interprogram communication: remote procedure calls [RPC]; Web services
G06F 40/30
Handling natural language data: semantic analysis

Family

Application63/033,496Us provisional application · filed 2 Jun 2020
US 2024/0249719 A1Pre-grant publication · published 25 Jul 2024
US 12,223,954 B2This patent

From the US filing history. Foreign family members aren’t listed.

Timeline

Citations

Open elsewhere

Building an agent or pulling many patents? Use the API or MCP server instead of scraping pages. Get an API key