# US 2019/0378510 A1 · User-programmable automated assistant

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0378510 A1
- Document ID: US20190378510A1
- Status: Published application
- Kind: A1
- Application number: 16/549,457
- Filed: 2019-08-23
- Published: 2019-12-12
- Earliest claimed: 2017-10-03
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Mihai Danila; Albry Smither
- Web page: https://patentgrep.com/patent/US20190378510A1

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06F 40/279: Recognition of textual entities
- G06F 40/35: Discourse or dialogue representation
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G10L 25/51: for comparison or discrimination
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L 2015/225: {Feedback of the input speech}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Techniques described herein relate to allowing users to employ voice-based human-to-computer dialog to program automated assistants with customized routines, or “dialog routines,” that can later be invoked to accomplish task(s). In various implementations, a first free form natural language input—that identifies a command to be mapped to a task and slot(s) required to be filled with values to fulfill the task—may be received from a user. A dialog routine may be stored that includes a mapping between the command and the task, and which accepts, as input, value(s) to fill the slot(s). Subsequent free form natural language input may be received from the user to (i) invoke the dialog routine based on the mapping, and/or (ii) to identify value(s) to fill the slot(s). Data indicative of at least the value(s) may be transmitted to a remote computing device for fulfillment of the task.

## Claims

### Claim 1

A method implemented by one or more processors, comprising:
receiving, from a user at one or more input components of a computing device, one or more speech inputs directed at an automated assistant executed by one or more of the processors, wherein the one or more speech inputs define a custom voice command and identify a task to be performed in response to receipt of the custom voice command by the automated assistant and one or more slots that are required to be filled with values in order to fulfill the task;
identifying, based on speech recognition output generated from the one or more speech inputs, the custom voice command, the task, and the one or more slots; and
creating and storing a custom dialog routine that includes a mapping between the custom voice command and the task, and which accepts, as input, one or more values to fill the one or more slots, wherein subsequent utterance of the custom command causes the automated assistant to engage in the custom dialog routine.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
receiving, by the automated assistant, a subsequent speech input, wherein the subsequent speech input includes the custom command;
identifying, based on subsequent speech recognition output generated from the subsequent speech input, the custom command; and
causing the automated assistant to engage in the custom dialog routine based on identifying the custom command in the subsequent speech input and based on the mapping.

### Claim 3 (depends on 2)

The method of claim 2, further comprising identifying, based on additional subsequent speech recognition output generated from the subsequent speech input or additional subsequent speech input, one or more values to be used to fill the one or more slots that are required to be filled with values in order to fulfill the task.

### Claim 4 (depends on 3)

The method of claim 3, wherein the task to which the command is mapped comprises a third-party agent task, and the method further comprises transmitting, to a remote computing device, data that is indicative of at least the one or more values to be used to fill the one or more slots, wherein the transmitting causes a third party agent executing on the remote computing device to fulfill the third-party agent task.

### Claim 5 (depends on 2)

The method of claim 2, further comprising generating, as part of engagement of the custom dialog routine by the automated assistant, natural language output that requests performance of the task based on one or more values provided to fill the one or more of the slots.

### Claim 6 (depends on 5)

The method of claim 5, further comprising transmitting the natural language output to a remote computing device as part of a simple messaging service (“SMS”) text message.

### Claim 7 (depends on 5)

The method of claim 5, further comprising transmitting the natural language output to a remote device to cause the remote device to audibly output the natural language output.

### Claim 8 (depends on 1)

The method of claim 1, wherein the one or more speech inputs include a slot-filling speech input from the user, wherein the slot-filling speech input includes a user-provided enumerated list of possible values to fill one or more of the slots.

### Claim 9

A system comprising one or more processors and memory storing instructions that, in response to execution of the instructions by the one or more processors, cause the one or more processors to perform the following operations
receive, from a user at one or more input components of a computing device, one or more speech inputs directed at an automated assistant executed by one or more of the processors, wherein the one or more speech inputs define a custom voice command and identify a task to be performed in response to receipt of the custom voice command by the automated assistant and one or more slots that are required to be filled with values in order to fulfill the task;
identify, based on speech recognition output generated from the one or more speech inputs, the custom voice command, the task, and the one or more slots; and
create and storing a custom dialog routine that includes a mapping between the custom voice command and the task, and which accepts, as input, one or more values to fill the one or more slots, wherein subsequent utterance of the custom command causes the automated assistant to engage in the custom dialog routine.

### Claim 10 (depends on 9)

The system of claim 9, further comprising instructions to:
receive, by the automated assistant, a subsequent speech input, wherein the subsequent speech input includes the custom command;
identify, based on subsequent speech recognition output generated from the subsequent speech input, the custom command; and
cause the automated assistant to engage in the custom dialog routine based on identifying the custom command in the subsequent speech input and based on the mapping.

### Claim 11 (depends on 10)

The system of claim 10, further comprising instructions to identify, based on additional subsequent speech recognition output generated from the subsequent speech input or additional subsequent speech input, one or more values to be used to fill the one or more slots that are required to be filled with values in order to fulfill the task.

### Claim 12 (depends on 11)

The system of claim 11, wherein the task to which the command is mapped comprises a third-party agent task, and the system further comprises instructions to transmit, to a remote computing device, data that is indicative of at least the one or more values to be used to fill the one or more slots, wherein the transmitting causes a third party agent executing on the remote computing device to fulfill the third-party agent task.

### Claim 13 (depends on 10)

The system of claim 10, further comprising instructions to generate, as part of engagement of the custom dialog routine by the automated assistant, natural language output that requests performance of the task based on one or more values provided to fill the one or more of the slots.

### Claim 14 (depends on 13)

The system of claim 13, further comprising instructions to transmit the natural language output to a remote computing device as part of a simple messaging service (“SMS”) text message.

### Claim 15 (depends on 13)

The system of claim 13, further comprising instructions to transmit the natural language output to a remote device to cause the remote device to audibly output the natural language output.

### Claim 16 (depends on 9)

The system of claim 9, wherein the one or more speech inputs include a slot-filling speech input from the user, wherein the slot-filling speech input includes a user-provided enumerated list of possible values to fill one or more of the slots.

### Claim 17

At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations:
receiving, from a user at one or more input components of a computing device, one or more speech inputs directed at an automated assistant executed by one or more of the processors, wherein the one or more speech inputs define a custom voice command and identify a task to be performed in response to receipt of the custom voice command by the automated assistant and one or more slots that are required to be filled with values in order to fulfill the task;
identifying, based on speech recognition output generated from the one or more speech inputs, the custom voice command, the task, and the one or more slots; and
creating and storing a custom dialog routine that includes a mapping between the custom voice command and the task, and which accepts, as input, one or more values to fill the one or more slots, wherein subsequent utterance of the custom command causes the automated assistant to engage in the custom dialog routine.

### Claim 18 (depends on 17)

The at least one non-transitory computer-readable medium of claim 17, further comprising instructions for:
receiving, by the automated assistant, a subsequent speech input, wherein the subsequent speech input includes the custom command;
identifying, based on subsequent speech recognition output generated from the subsequent speech input, the custom command; and
causing the automated assistant to engage in the custom dialog routine based on identifying the custom command in the subsequent speech input and based on the mapping.

### Claim 19 (depends on 18)

The at least one non-transitory computer-readable medium of claim 18, further comprising instructions for identifying, based on additional subsequent speech recognition output generated from the subsequent speech input or additional subsequent speech input, one or more values to be used to fill the one or more slots that are required to be filled with values in order to fulfill the task.

### Claim 20 (depends on 19)

The at least one non-transitory computer-readable medium of claim 19, wherein the task to which the command is mapped comprises a third-party agent task, and the at least one non-transitory computer-readable medium further comprises instructions for transmitting, to a remote computing device, data that is indicative of at least the one or more values to be used to fill the one or more slots, wherein the transmitting causes a third party agent executing on the remote computing device to fulfill the third-party agent task.
