# US 12,182,767 B1 · Complex task cognitive planning and execution system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,182,767 B1
- Document ID: US12182767B1
- Status: Granted
- Kind: B1
- Application number: 15/843,891
- Filed: 2017-12-15
- Granted: 2024-12-31
- Earliest claimed: 2016-12-22
- Assignees: Brain Technologies, Inc.
- Applicants: Brain Technologies, Inc.
- Inventors: Sheng Yue; Yuan Lin
- Web page: https://patentgrep.com/patent/US12182767B1

## CPC

- G06F 9/541: {via adapters, e.g. between incompatible applications}
- G06F 9/542: {Event management; Broadcasting; Multicasting; Notifications}
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 9/453: {Help systems}
- G06F 16/168: {Details of user interfaces specifically adapted to file systems, e.g. browsing and visualisation, 2d or 3d GUIs (query results presentation G06F16/156)}
- G06F 16/1734: {Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs}
- G06F 16/9024: {Graphs; Linked lists (G06F16/9027 takes precedence)}
- G06F 16/951: Indexing; Web crawling techniques
- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G06N 20/00: Machine learning
- G06Q 10/1093: Calendar-based scheduling for persons or groups
- G06F 2209/544: Remote
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Disclosed is a system for determining sequences of operations that will automatically execute one or more tasks specified by a user. In some embodiments, the sequences of operations are based on operations that have been previously performed by users and recorded by the system. The system interprets an intention of a user based on analysis of terms used by the user to indicate a request. The system generates a sequence of operations, executable by an operating system associated with a client device that will perform one or more tasks specified or implied by the request of the user.

## Claims

### Claim 1

A computer-implemented method comprising:
training, by a remote computing system, an operation machine learning model based on training samples of sequences of operations responding to user requests of a plurality of users, wherein training the operation machine learning model comprises:
recording the sequences of operations occurring at client devices controlled by the plurality of users, the sequence of operations directed by the plurality of users to use a plurality of third-party software applications at the client devices, each sequence of operations comprising (1) an input specified by one of the plurality of users and (2) a software application of the plurality of third-party software applications that is selected by said one of the plurality of users to be executed at a client device to perform a task associated with the input;
collecting the sequences of operations from the plurality of users as the training samples, the training samples comprising a plurality of tasks and recordings of how the plurality of users executing third-party software applications at the client devices to perform the plurality of tasks;
generating an operation graph that comprises a plurality of paths connecting the plurality of tasks and the plurality of third-party software applications to be executed; and
training the operation machine learning model by adjusting the operation graph based on the training samples;
receiving, by the remote computing system, a request from a particular client device of a particular user;
receiving, by the remote computing system, authorizations from the particular user of executions of a list of third-party software applications that are linked to the particular user;
determining, based on the request, a set of tasks to complete to respond to the request, wherein determining the set of tasks comprises applying the operation machine learning model to the request to identify that two or more third-party software applications in the list authorized by the particular user are to be executed, wherein applying the operation machine learning model comprises:
for each task in the set of tasks:
identifying, from the operation graph, a path of the plurality of paths that connects the task with a particular third-party software application to be executed,
wherein the operation graph comprises a plurality of nodes and a plurality of directed edges connecting the plurality of nodes,
the plurality of nodes represents a plurality of view-structures that describe components of user interfaces of the plurality of third-party software applications,
each directed edge represents a user interaction with a first view-structure that prompts a user interface transitioning from the first view-structure to a second view-structure, and the path is associated with a sequence order to execute the task,
the sequence order comprising executing the particular third-party software application that is connected to the task by the path;
configuring an overall sequence of operations by combining each of the tasks in the set of tasks that are determined by the operation machine learning model, the overall sequence of operations comprising switching between two or more third-party software applications that are distinct from each other;
performing the overall sequence of operations that is based on the request without further user input, wherein at least a portion of the overall sequence of operations is performed by the remote computing system, the overall sequence of operation comprising executing the switching between two or more third-party software applications that are distinct from each other, the two or more distinct third-party software applications being in the list authorized by the particular user;
causing the particular client device to display icons of the two or more third-party software applications that are executed as part of the overall sequence operation;
and causing the particular client device to display a result of the overall sequence of operations that are at least partially performed by the remote computing system and that include the switching between two or more third-party software applications.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein configuring the overall sequence of operations further comprises:
identifying main terms from the request in a category store;
determining a domain of the main term based on a mapping of terms to domains in a mapping of the category store;
identifying in the operation graph, a sequence of operations, the sequence of operations associated with the determined domain; and
selecting one or more operations that, when executed in sequence, will perform the set of tasks.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein the one or more operations are selected from the operation graph based on terms from the request that are associated with one or more of the operations as stored in the operation graph.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the request is received in a natural language format.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein determining the set of tasks to complete to respond to the request comprises:
parsing the request into component parts of speech;
identifying a set of main terms from the request, the main terms comprising nouns and verbs;
identifying a set of detailed terms from the request, the detailed terms comprising adjectives and attributes; and
determining one or more groupings of the main terms, each grouping of the one or more groupings indicative of one or more operations that may be performed.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, further comprising: initializing content within a view-structure associated with an operation of the overall sequence of operations.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, further comprising:
associating metadata with each operation in the overall sequence of operations, the metadata including instructions about whether an operating system should render a user interface associated with each operation on a screen of the particular client device when the operation executes.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, further comprising:
providing a list of the determined set of tasks for display to the particular user at particular client device; and
receiving, from the client device, a user selection of one of the tasks.

### Claim 9

A computer system comprising:
one or more computer processors for executing computer program instructions;
and a non-transitory computer-readable storage medium comprising stored instructions executable by the one or more computer processors to:
train, by a remote computing system, an operation machine learning model based on training samples of sequences of operations responding to user requests of a plurality of users, wherein training the operation machine learning model comprises:
recording the sequences of operations occurring at client devices controlled by the plurality of users, the sequence of operations directed by the plurality of users to use a plurality of third-party software applications at the client devices, each sequence of operations comprising (1) an input specified by one of the plurality of users and (2) a software application of the plurality of third-party software applications that is selected by said one of the plurality of users to be executed at a client device to perform a task associated with the input;
collecting the sequences of operations from the plurality of users as the training samples, the training samples comprising a plurality of tasks and recordings of how the plurality of users executing third-party software applications at the client devices to perform the plurality of tasks;
generating an operation graph that comprises a plurality of paths connecting the plurality of tasks and the plurality of third-party software applications to be executed; and
training the operation machine learning model by adjusting the operation graph based on the training samples;
receive, by the remote computing system, a request from a particular client device of a particular user;
receive, by the remote computing system, authorizations from the particular user of executions of a list of third-party software applications that are linked to the particular user;
determine, based on the request, a set of tasks to complete to respond to the request, wherein determining the set of tasks comprises applying the operation machine learning model to the request to identify that two or more third-party software applications in the list authorized by the particular user are to be executed, wherein applying the operation machine learning model comprises:
for each task in the set of tasks:
identify, from the operation graph, a path of the plurality of paths that connects the task with a particular third-party software application to be executed,
wherein the operation graph comprises a plurality of nodes and a plurality of directed edges connecting the plurality of nodes,
the plurality of nodes represents a plurality of view-structures that describe components of user interfaces of the plurality of third-party software applications,
each directed edge represents a user interaction with a first view-structure that prompts a user interface transitioning from the first view-structure to a second view-structure,
and the path is associated with a sequence order to execute the task, the sequence order comprising executing the particular third-party software application that is connected to the task by the path;
configure an overall sequence of operations by combining each of the tasks in the set of tasks that are determined by the operation machine learning model, the overall sequence of operations comprising switching between two or more third-party software applications that are distinct from each other;
perform the overall sequence of operations that is based on the request without further user input, wherein at least a portion of the overall sequence of operations is performed by the remote computing system, the overall sequence of operation comprising executing the switching between two or more third-party software applications that are distinct from each other, the two or more distinct third-party software applications being in the list authorized by the particular user;
cause the particular client device to display icons of the two or more third-party software applications that are executed as part of the overall sequence operation;
and cause the particular client device to display a result of the overall sequence of operations that are at least partially performed by the remote computing system and that include the switching between two or more third-party software applications.

### Claim 10 (depends on 9)

The computer system of claim 9, wherein the instruction to configure the overall sequence of operations further comprises instructions that when executed causes the processor to:
identify main terms from the request in a category store;
determine a domain of the main term based on a mapping of terms to domains in a mapping of the category store;
identify in the operation graph, a sequence of operations, the sequence of operations associated with the determined domain; and
select one or more operations that, when executed in sequence, will perform the set of tasks.

### Claim 11 (depends on 10)

The computer system of claim 10, further comprises instructions that when executed causes the processor to select the one or more operations from the operation graph based on terms from the request that are associated with one or more of the operations as stored in the operation graph.

### Claim 12 (depends on 9)

The computer system of claim 9, further comprising instructions that when executed causes the processor to receive requests in a natural language format.

### Claim 13 (depends on 9)

The computer system of claim 9, the instructions to determine the set of tasks to complete to respond to the request further comprises instructions that when executed causes the processor to:
parse the request into component parts of speech;
identify a set of main terms from the request, the main terms comprising nouns and verbs;
identify a set of detailed terms from the request, the detailed terms comprising adjectives and attributes; and
determine one or more groupings of the main terms, each grouping of the one or more groupings indicative of one or more operations that may be performed.

### Claim 14 (depends on 9)

The computer system of claim 9, wherein the instructions when executed causes the processor to initialize content within a view-structure associated with an operation.

### Claim 15 (depends on 9)

The computer system of claim 9, further comprises instructions that when executed causes the processor to:
associate metadata with each operation in the overall sequence of operations, the metadata including instructions about whether an operating system should render a user interface associated with each operation on a screen of the particular client device when the operation executes.

### Claim 16 (depends on 9)

The computer system of claim 9, further comprises instructions that when executed causes the processor to:
provide a list of the determined set of tasks for display to the particular user at the particular client device; and
receive, from the client device, a user selection of one of the tasks.

### Claim 17

A non-transitory computer-readable medium comprising stored instructions that when executed by one or more processors of one or more computing devices, cause the one or more computing devices to:
train, by a remote computing system, an operation machine learning model based on training samples of sequences of operations responding to user requests of a plurality of users, wherein training the operation machine learning model comprises:
recording the sequences of operations occurring at client devices controlled by the plurality of users, the sequence of operations directed by the plurality of users to use a plurality of third-party software applications at the client devices, each sequence of operations comprising (1) an input specified by one of the plurality of users and (2) a software application of the plurality of third-party software applications that is selected by said one of the plurality of users to be executed at a client device to perform a task associated with the input;
collecting the sequences of operations from the plurality of users as the training samples, the training samples comprising a plurality of tasks and recordings of how the plurality of users executing third-party software applications at the client devices to perform the plurality of tasks;
generating an operation graph that comprises a plurality of paths connecting the plurality of tasks and the plurality of third-party software applications to be executed; and
training the operation machine learning model by adjusting the operation graph based on the training samples;
receive, by the remote computing system, a request from a particular client device of a particular user;
receive, by the remote computing system, authorizations from the particular user of executions of a list of third-party software applications that are linked to the particular user;
determine, based on the request, a set of tasks to complete to respond to the request, wherein determining the set of tasks comprises applying the operation machine learning model to the request to identify that two or more third-party software applications in the list authorized by the particular user are to be executed, wherein applying the operation machine learning model comprises:
for each task in the set of tasks:
identify, from the operation graph, a path of the plurality of paths that connects the task with a particular third-party software application to be executed,
wherein the operation graph comprises a plurality of nodes and a plurality of directed edges connecting the plurality of nodes,
the plurality of nodes represents a plurality of view-structures that describe components of user interfaces of the plurality of third-party software applications,
each directed edge represents a user interaction with a first view-structure that prompts a user interface transitioning from the first view-structure to a second view-structure, and the path is associated with a sequence order to execute the task,
the sequence order comprising executing the particular third-party software application that is connected to the task by the path;
configure an overall sequence of operations by combining each of the tasks in the set of tasks that are determined by the operation machine learning model, the overall sequence of operations comprising switching between two or more third-party software applications that are distinct from each other;
perform the overall sequence of operations that is based on the request without further user input, wherein at least a portion of the overall sequence of operations is performed by the remote computing system, the overall sequence of operation comprising executing the switching between two or more third-party software applications that are distinct from each other, the two or more distinct third-party software applications being in the list authorized by the particular user;
cause the particular client device to display icons of the two or more third-party software applications that are executed as part of the overall sequence operation; and
cause the particular client device to display a result of the overall sequence of operations that are at least partially performed by the remote computing system and that include the switching between two or more third-party software applications.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium of claim 17, the instructions to determine the set of tasks to complete to respond to the request further comprises instructions that when executed causes the processor to:
parse the request into component parts of speech;
identify a set of main terms from the request, the main terms comprising nouns and verbs;
identify a set of detailed terms from the request, the detailed terms comprising adjectives and attributes; and
determine one or more groupings of the main terms, each grouping of the one or more groupings indicative of one or more operations that may be performed.

### Claim 19 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein the instruction to configure the overall sequence of operations further comprises instructions that when executed causes the processor to:
identify main terms from the request in a category store;
determine a domain of the main term based on a mapping of terms to domains in a mapping of the category store;
identify in the operation graph, a sequence of operations, the sequence of operations associated with the determined domain; and
select one or more operations that, when executed in sequence, will perform the set of tasks.

### Claim 20 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein the instructions t when executed causes the processor to initialize content within a view-structure associated with an operation.