# US 12,002,463 B2 · Systems and methods for voice-based initiation of custom device actions

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,463 B2
- Document ID: US12002463B2
- Status: Granted
- Kind: B2
- Application number: 17/728,614
- Filed: 2022-04-25
- Published: 2022-08-04
- Granted: 2024-06-04
- Earliest claimed: 2018-03-07
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Bo Wang; Venkat Kotla; Chad Yoshikawa; Chris Ramsdale; Pravir Gupta; Alfonso Gomez-Jordana; Kevin Yeun; Jae Won Seo; Lantian Zheng; Sang Soo Sung
- Web page: https://patentgrep.com/patent/US12002463B2

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 16/632: Query formulation
- G06F 16/634: {Query by example, e.g. query by humming}
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G10L 2015/088: {Word spotting}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems and methods for enabling voice-based interactions with electronic devices can include a data processing system maintaining a plurality of device action data sets and a respective identifier for each device action data set. The data processing system can receive, from an electronic device, an audio signal representing a voice query and an identifier. The data processing system can identify, using the identifier, a device action data set. The data processing system can identify a device action from device action data set based on content of the audio signal. The data processing system can then identify, from the device action dataset, a command associated with the device action and send the command to the for execution device for execution.

## Claims

### Claim 1

A data processing system to provide content responsive to voice-based interactions, comprising:
a memory to store device action data including a plurality of device action-command pairs supported by a plurality of electronic devices associated with a device model, each device action-command pair including a respective device action of a plurality of device actions and a respective device executable command of a plurality of device executable commands to trigger performance of the respective device action, the respective device executable command being an executable command specific to the plurality of electronic devices associated with the device model;
a device action customization component to map a device model identifier indicative of the device model of the plurality of electronic devices to each of the plurality of device action-command pairs supported by the plurality of electronic devices associated with the device model;
a communications interface to receive, from an electronic device, an audio signal and the device model identifier, the audio signal obtained by the electronic device responsive to a voice-based query;
a natural language processor component to identify, using content associated with the audio signal and the device model identifier, a device action-command pair of the plurality of device action-command pairs;
the device action customization component to identify a context of the voice-based query based on the device action data or the device action-command pair;
a content selector component to select a digital component based on the context of the voice-based query, the digital component comprising audio or visual content; and
the communications interface to transmit the digital component and a device executable command associated with the device action-command pair to the electronic device, the device executable command to cause performance of the device action associated with the device action-command pair and the digital component for presentation by the electronic device, the digital component causing the electronic device to render the audio or visual content on the electronic device in connection with and based on the performance of the device action.

### Claim 2 (depends on 1)

The data processing system of claim 1, wherein the device action is a first device action and the data processing system comprising:
the device action customization component to:
identify a predefined sequence of device actions associated with the voice-based query using the first device action associated with the device-action command pair;
identify one or more second device actions following the first device action associated with the device-action command pair in the predefined sequence of device actions; and
identify an attribute of the context of the voice-based query based on the one or more second device actions following the device action.

### Claim 3 (depends on 1)

The data processing system of claim 1, wherein the digital component includes an audio digital component.

### Claim 4 (depends on 1)

The data processing system of claim 1, comprising:
the memory to maintain a plurality of responses for serving to the plurality of electronic devices, each response of the plurality of responses mapped to a corresponding device-action pair of the plurality of device-action pairs supported by the plurality of electronic devices;
the device action customization component to identify a response of the plurality of responses corresponding to the device action-command pair; and
the communications interface to transmit the response to the electronic device for presentation.

### Claim 5 (depends on 1)

The data processing system of claim 1, comprising:
the memory to maintain, for each device action-command pair of the plurality of device action-command pairs, one or more corresponding keywords; and
the device action customization component to identify the context of the voice-based query based on the one or more keywords associated with the device action-command pair identified by the natural language processor component.

### Claim 6 (depends on 1)

The data processing system of claim 1, wherein the plurality of device action-command pairs are specific to the device model.

### Claim 7 (depends on 6)

The data processing system of claim 6 comprising:
the memory to maintain a description of the plurality of electronic devices associated with device model; and
the device action customization component to identify an attribute of the context of the voice-based query based on the description of the plurality of electronic devices associated with device model.

### Claim 8 (depends on 1)

The data processing system of claim 1, wherein the device-model identifier is associated with an application installed on the plurality of electronic devices and the plurality of device action-command pairs are specific to the application.

### Claim 9 (depends on 8)

The data processing system of claim 8 comprising:
the memory to maintain a description of the application installed on the plurality of electronic devices; and
the device action customization component to identify an attribute of the context of the voice-based query based on the description of the application installed on the plurality of electronic devices.

### Claim 10 (depends on 8)

The data processing system of claim 8 comprising:
the device action customization component to identify one or more parameter values of the device executable command associated with the device action-command pair; and
the device action customization component to identify an attribute of the context of the voice-based query based on the one or more parameter values of the device executable command associated with the device action-command pair.

### Claim 11 (depends on 1)

The data processing system of claim 1, comprising:
the device action customization component to provide a user interface to allow transmission of the device action data and the device-model identifier to the data processing system.

### Claim 12 (depends on 1)

The data processing system of claim 1, comprising:
the device action customization component to provide a restful application programming interface (API) to allow transmission of the device action data and the device-model identifier to the data processing system.

### Claim 13

A method of providing content responsive to voice-based interactions, the method comprising:
storing, in a memory, device action data including a plurality of device action-command pairs supported by a plurality of electronic devices associated with a device model, each device action-command pair including a respective device action of a plurality of device actions and a respective device executable command of a plurality of device executable commands to trigger performance of the respective device action, the respective device executable command being an executable command specific to the plurality of electronic devices associated with the device model;
mapping a device model identifier indicative of the device model of the plurality of electronic devices to each of the plurality of device action-command pairs supported by the plurality of electronic devices associated with the device model;
receiving, from an electronic device, an audio signal and the device model identifier, the audio signal obtained by the electronic device responsive to a voice-based query;
identifying, using content associated with the audio signal and the device model identifier, a device action-command pair of the plurality of device action-command pairs;
identifying a context of the voice-based query based on the device action data or the device action-command pair;
selecting a digital component based on the context of the voice-based query, the digital component comprising audio or visual content; and
transmitting the digital component and a device executable command associated with the device action-command pair to the electronic device, the device executable command to cause performance of the device action associated with the device action-command pair and the digital component for presentation by the electronic device, the digital component causing the electronic device to render the audio or visual content on the electronic device in connection with and based on the performance of the device action.

### Claim 14 (depends on 13)

The method of claim 13, comprising:
maintaining, for each device action-command pair of the plurality of device action-command pairs, one or more corresponding keywords; and
identifying the context of the voice-based query based on the one or more keywords associated with the device action-command pair identified by the natural language processor component.

### Claim 15 (depends on 13)

The method of claim 13, wherein the plurality of device action-command pairs are specific to the device model.

### Claim 16 (depends on 15)

The method of claim 15 comprising:
maintaining a description of the plurality of electronic devices associated with device model; and
identifying an attribute of the context of the voice-based query based on the description of the plurality of electronic devices associated with device model.

### Claim 17 (depends on 13)

The method of claim 13, wherein the device model identifier is associated with an application installed on the plurality of electronic devices and the plurality of device action-command pairs are specific to application.

### Claim 18 (depends on 17)

The method of claim 17 comprising:
maintaining a description of the application installed on the plurality of electronic devices; and
identifying an attribute of the context of the voice-based query based on the description of the application installed on the plurality of electronic devices.

### Claim 19 (depends on 17)

The method of claim 17 comprising:
identifying one or more parameter values of the device executable command associated with the device action-command pair; and
identifying an attribute of the context of the voice-based query based on the one or more parameter values of the device executable command associated with the device action-command pair.

### Claim 20

A method of providing content responsive to voice-based interactions, the method comprising:
storing, in a memory, device action data including a plurality of device action-command pairs supported by a plurality of electronic devices, each device action-command pair including a respective device action of a plurality of device actions and a respective device executable command of a plurality of device executable commands to trigger performance of the respective device action;
mapping an identifier to each of the plurality of device action-command pairs supported by the plurality of electronic devices;
receiving, from an electronic device, an audio signal and the identifier, the audio signal obtained by the electronic device responsive to a voice-based query;
identifying, using content associated with the audio signal and the identifier, a device action-command pair of the plurality of device action-command pairs, wherein the device action associated with the device action-command pair is a first device action;
identifying a predefined sequence of device actions associated with the voice-based query using the first device action associated with the device-action command pair;
identifying one or more second device actions following the first device action associated with the device-action command pair in the predefined sequence of device actions;
identifying a context of the voice-based query based on the device action data or the device action-command pair;
identifying an attribute of the context of the voice-based query based on the one or more second device actions following the device action;
selecting a digital component based on the context of the voice-based query; and
transmitting the digital component and a device executable command associated with the device action-command pair to the electronic device, the device executable command to cause performance of the device action associated with the device action-command pair and the digital component for presentation by the electronic device.