Back to results
US 12,182,767 B1Granted
Complex task cognitive planning and execution system
AssigneeBrain Technologies, Inc.
InventorsSheng Yue, Yuan Lin
Life of this patent
Estimating the term…
Abstract
Disclosed is a system for determining sequences of operations that will automatically execute one or more tasks specified by a user. In some embodiments, the sequences of operations are based on operations that have been previously performed by users and recorded by the system. The system interprets an intention of a user based on analysis of terms used by the user to indicate a request. The system generates a sequence of operations, executable by an operating system associated with a client device that will perform one or more tasks specified or implied by the request of the user.
Claims20
A computer-implemented method comprising:
- training, by a remote computing system, an operation machine learning model based on training samples of sequences of operations responding to user requests of a plurality of users, wherein training the operation machine learning model comprises:
- recording the sequences of operations occurring at client devices controlled by the plurality of users, the sequence of operations directed by the plurality of users to use a plurality of third-party software applications at the client devices, each sequence of operations comprising (1) an input specified by one of the plurality of users and (2) a software application of the plurality of third-party software applications that is selected by said one of the plurality of users to be executed at a client device to perform a task associated with the input;
- collecting the sequences of operations from the plurality of users as the training samples, the training samples comprising a plurality of tasks and recordings of how the plurality of users executing third-party software applications at the client devices to perform the plurality of tasks;
- generating an operation graph that comprises a plurality of paths connecting the plurality of tasks and the plurality of third-party software applications to be executed;
- and
- training the operation machine learning model by adjusting the operation graph based on the training samples;
- receiving, by the remote computing system, a request from a particular client device of a particular user;
- receiving, by the remote computing system, authorizations from the particular user of executions of a list of third-party software applications that are linked to the particular user;
- determining, based on the request, a set of tasks to complete to respond to the request, wherein determining the set of tasks comprises applying the operation machine learning model to the request to identify that two or more third-party software applications in the list authorized by the particular user are to be executed, wherein applying the operation machine learning model comprises:
- for each task in the set of tasks:
- identifying, from the operation graph, a path of the plurality of paths that connects the task with a particular third-party software application to be executed,
- wherein the operation graph comprises a plurality of nodes and a plurality of directed edges connecting the plurality of nodes,
- the plurality of nodes represents a plurality of view-structures that describe components of user interfaces of the plurality of third-party software applications,
- each directed edge represents a user interaction with a first view-structure that prompts a user interface transitioning from the first view-structure to a second view-structure, and the path is associated with a sequence order to execute the task,
- the sequence order comprising executing the particular third-party software application that is connected to the task by the path;
- configuring an overall sequence of operations by combining each of the tasks in the set of tasks that are determined by the operation machine learning model, the overall sequence of operations comprising switching between two or more third-party software applications that are distinct from each other;
- performing the overall sequence of operations that is based on the request without further user input, wherein at least a portion of the overall sequence of operations is performed by the remote computing system, the overall sequence of operation comprising executing the switching between two or more third-party software applications that are distinct from each other, the two or more distinct third-party software applications being in the list authorized by the particular user;
- causing the particular client device to display icons of the two or more third-party software applications that are executed as part of the overall sequence operation;
- and causing the particular client device to display a result of the overall sequence of operations that are at least partially performed by the remote computing system and that include the switching between two or more third-party software applications.
16 more elements
A computer system comprising:
- one or more computer processors for executing computer program instructions;
- and a non-transitory computer-readable storage medium comprising stored instructions executable by the one or more computer processors to:
- train, by a remote computing system, an operation machine learning model based on training samples of sequences of operations responding to user requests of a plurality of users, wherein training the operation machine learning model comprises:
- recording the sequences of operations occurring at client devices controlled by the plurality of users, the sequence of operations directed by the plurality of users to use a plurality of third-party software applications at the client devices, each sequence of operations comprising (1) an input specified by one of the plurality of users and (2) a software application of the plurality of third-party software applications that is selected by said one of the plurality of users to be executed at a client device to perform a task associated with the input;
- collecting the sequences of operations from the plurality of users as the training samples, the training samples comprising a plurality of tasks and recordings of how the plurality of users executing third-party software applications at the client devices to perform the plurality of tasks;
- generating an operation graph that comprises a plurality of paths connecting the plurality of tasks and the plurality of third-party software applications to be executed;
- and
- training the operation machine learning model by adjusting the operation graph based on the training samples;
- receive, by the remote computing system, a request from a particular client device of a particular user;
- receive, by the remote computing system, authorizations from the particular user of executions of a list of third-party software applications that are linked to the particular user;
- determine, based on the request, a set of tasks to complete to respond to the request, wherein determining the set of tasks comprises applying the operation machine learning model to the request to identify that two or more third-party software applications in the list authorized by the particular user are to be executed, wherein applying the operation machine learning model comprises:
- for each task in the set of tasks:
- identify, from the operation graph, a path of the plurality of paths that connects the task with a particular third-party software application to be executed,
- wherein the operation graph comprises a plurality of nodes and a plurality of directed edges connecting the plurality of nodes,
- the plurality of nodes represents a plurality of view-structures that describe components of user interfaces of the plurality of third-party software applications,
- each directed edge represents a user interaction with a first view-structure that prompts a user interface transitioning from the first view-structure to a second view-structure,
- and the path is associated with a sequence order to execute the task, the sequence order comprising executing the particular third-party software application that is connected to the task by the path;
- configure an overall sequence of operations by combining each of the tasks in the set of tasks that are determined by the operation machine learning model, the overall sequence of operations comprising switching between two or more third-party software applications that are distinct from each other;
- perform the overall sequence of operations that is based on the request without further user input, wherein at least a portion of the overall sequence of operations is performed by the remote computing system, the overall sequence of operation comprising executing the switching between two or more third-party software applications that are distinct from each other, the two or more distinct third-party software applications being in the list authorized by the particular user;
- cause the particular client device to display icons of the two or more third-party software applications that are executed as part of the overall sequence operation;
- and cause the particular client device to display a result of the overall sequence of operations that are at least partially performed by the remote computing system and that include the switching between two or more third-party software applications.
18 more elements
A non-transitory computer-readable medium comprising stored instructions that when executed by one or more processors of one or more computing devices, cause the one or more computing devices to:
- train, by a remote computing system, an operation machine learning model based on training samples of sequences of operations responding to user requests of a plurality of users, wherein training the operation machine learning model comprises:
- recording the sequences of operations occurring at client devices controlled by the plurality of users, the sequence of operations directed by the plurality of users to use a plurality of third-party software applications at the client devices, each sequence of operations comprising (1) an input specified by one of the plurality of users and (2) a software application of the plurality of third-party software applications that is selected by said one of the plurality of users to be executed at a client device to perform a task associated with the input;
- collecting the sequences of operations from the plurality of users as the training samples, the training samples comprising a plurality of tasks and recordings of how the plurality of users executing third-party software applications at the client devices to perform the plurality of tasks;
- generating an operation graph that comprises a plurality of paths connecting the plurality of tasks and the plurality of third-party software applications to be executed;
- and
- training the operation machine learning model by adjusting the operation graph based on the training samples;
- receive, by the remote computing system, a request from a particular client device of a particular user;
- receive, by the remote computing system, authorizations from the particular user of executions of a list of third-party software applications that are linked to the particular user;
- determine, based on the request, a set of tasks to complete to respond to the request, wherein determining the set of tasks comprises applying the operation machine learning model to the request to identify that two or more third-party software applications in the list authorized by the particular user are to be executed, wherein applying the operation machine learning model comprises:
- for each task in the set of tasks:
- identify, from the operation graph, a path of the plurality of paths that connects the task with a particular third-party software application to be executed,
- wherein the operation graph comprises a plurality of nodes and a plurality of directed edges connecting the plurality of nodes,
- the plurality of nodes represents a plurality of view-structures that describe components of user interfaces of the plurality of third-party software applications,
- each directed edge represents a user interaction with a first view-structure that prompts a user interface transitioning from the first view-structure to a second view-structure, and the path is associated with a sequence order to execute the task,
- the sequence order comprising executing the particular third-party software application that is connected to the task by the path;
- configure an overall sequence of operations by combining each of the tasks in the set of tasks that are determined by the operation machine learning model, the overall sequence of operations comprising switching between two or more third-party software applications that are distinct from each other;
- perform the overall sequence of operations that is based on the request without further user input, wherein at least a portion of the overall sequence of operations is performed by the remote computing system, the overall sequence of operation comprising executing the switching between two or more third-party software applications that are distinct from each other, the two or more distinct third-party software applications being in the list authorized by the particular user;
- cause the particular client device to display icons of the two or more third-party software applications that are executed as part of the overall sequence operation;
- and
- cause the particular client device to display a result of the overall sequence of operations that are at least partially performed by the remote computing system and that include the switching between two or more third-party software applications.
17 more elements
Dependencies are inferred from the claim text.