# US 2018/0322878 A1 · Multimodal transmission of packetized data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0322878 A1
- Document ID: US20180322878A1
- Status: Published application
- Kind: A1
- Application number: 16/039,202
- Filed: 2018-07-18
- Published: 2018-11-08
- Earliest claimed: 2016-12-30
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Gaurav Bhaya; Robert Stets
- Web page: https://patentgrep.com/patent/US20180322878A1

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G10L 15/14: using statistical models, e.g. Hidden Markov Models [HMMs] (G10L15/18 takes precedence)
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- H04L 47/25: with rate being modified by the source upon detecting a change of network conditions
- G10L 2015/088: {Word spotting}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A system of multi-modal transmission of packetized data in a voice activated data packet based computer network environment is provided. A natural language processor component can parse an input audio signal to identify a request and a trigger keyword. Based on the input audio signal, a direct action application programming interface can generate a first action data structure, and a content selector component can select a content item. An interface management component can identify first and second candidate interfaces, and respective resource utilization values. The interface management component can select, based on the resource utilization values, the first candidate interface to present the content item. The interface management component can provide the first action data structure to the client computing device for rendering as audio output, and can transmit the content item converted for a first modality to deliver the content item for rendering from the selected interface.

## Claims

### Claim at position 1

-20. (canceled)

### Claim 21

A system of multi-modal transmission of packetized data in a voice activated data packet based computer network environment, comprising:
a natural language processor component executed by a data processing system to receive, via an interface of the data processing system, data packets comprising an input audio signal detected by a sensor of a client computing device;
the natural language processor component to parse the input audio signal to identify a request;
a direct action application programming interface of the data processing system to generate, based the request, a first action data structure;
a content selector component executed by the data processing system to receive at least the request to select a content item;
an interface management component of the data processing system to:
poll a plurality of interfaces to identify a first candidate interface and a second candidate interface;
determine a first interface parameter for the first candidate interface and a second interface parameter for the second candidate interface;
select, based on a comparison of the first interface parameter and the second interface parameter, the first candidate interface as a selected interface to present the content item;
provide the first action data structure to the client computing device for rendering as audio output from the client computing device; and
transmit the content item for rendering at the selected interface.

### Claim 22 (depends on 21)

The system of claim 21, comprising:
the natural language processor component to parse the input audio signal to identify a trigger keyword corresponding to the request; and
the content selector component to select the content item based at least on the trigger keyword.

### Claim 23 (depends on 21)

The system of claim 21, wherein first candidate interface is an interface of a second client device.

### Claim 24 (depends on 21)

The system of claim 21, wherein first candidate interface and second candidate interface are two different interfaces of the client computing device.

### Claim 25 (depends on 24)

The system of claim 24, comprising:
the interface management component to provide the first action data structure for rendering from the second candidate interface.

### Claim 26 (depends on 21)

The system of claim 21, comprising the interface management component to:
determine a first resource utilization value for the first candidate interface based on the first interface parameter and a second resource utilization value for the second candidate interface based the second interface parameter and at least one of a battery status, a processor utilization, a memory utilization, or a network bandwidth utilization; and
select the first candidate interface as the selected interface to present the content item based on the first resource utilization value and the second resource utilization value.

### Claim 27 (depends on 26)

The system of claim 26, comprising the interface management component to:
convert the content item for delivery in a modality compatible with the selected interface selected based on the comparison of the first resource utilization value and the second resource utilization value.

### Claim 28 (depends on 21)

The system of claim 21, comprising:
the interface management component to deliver the content item to the client computing device via the selected interface subsequent to transmission of the first action data structure to the client computing device.

### Claim 29 (depends on 21)

The system of claim 21, the interface management component to:
determine that the first candidate interface corresponds to the client computing device linked to an account, and that the second candidate interface corresponds to a second client computing device linked to the account.

### Claim 30 (depends on 21)

The system of claim 21, wherein the first candidate interface includes a display screen, and the second candidate interface includes an audio interface.

### Claim 31 (depends on 21)

The system of claim 21, comprising the interface management component to:
identify a third candidate interface;
determine a third interface parameter for the third candidate interface; and
assign a hierarchy to the first candidate interface, the second candidate interface, and the third candidate interface based on the first interface parameter, the second interface parameter, and the third interface parameter.

### Claim 32 (depends on 31)

The system of claim 31, comprising:
the interface management component to deliver the content item to the client computing device via the selected interface based on the hierarchy.

### Claim 33 (depends on 21)

The system of claim 21, wherein the plurality of interfaces include at least one of a display screen, an audio interface, a vibration interface, an email interface, a push notification interface, a mobile computing device interface, a portable computing device application, a content slot on an online document, a chat application, mobile computing device application, a laptop, a watch, a virtual reality headset, and a speaker.

### Claim 34 (depends on 21)

The system of claim 21, wherein the client computing device is a first client computing device, comprising the interface management component to:
determine a distance between the first client computing device and a second client computing device; and to
transmit the content item to the second client computing device.

### Claim 35 (depends on 21)

The system of claim 21, wherein the client computing device is a first client computing device, comprising the interface management component to:
transmit the content item to the first client computing device within a pre-determined time period from receipt of the input audio signal by the natural language processor component.

### Claim 36 (depends on 21)

The system of claim 21, comprising:
the interface management component to determine, from the second interface parameter, that the second candidate interface is unavailable.

### Claim 37 (depends on 21)

The system of claim 21, comprising:
provide the first action data structure to the client computing device for rendering with an indication of the content item, the indication identifying the first candidate interface.

### Claim 38

A method of multi-modal transmission of packetized data in a voice activated data packet based computer network environment, comprising:
receiving, by a natural language processor component of a data processing system, via an interface of the data processing system, data packets comprising an input audio signal detected by a sensor of a client computing device;
parsing, by the natural language processor component, the input audio signal to identify a request;
generating, by a direct action application programming interface of the data processing system and based on the request, a first action data structure;
selecting, by a content selector component of the data processing system, a content item based on the request;
polling, by an interface management component of the data processing system, a plurality of interfaces to identify a first candidate interface and a second candidate interface;
determining, by the interface management component of the data processing system, a first interface parameter for the first candidate interface and a second interface parameter for the second candidate interface;
selecting, by the interface management component of the data processing system and based on a comparison of the first interface parameter and the second interface parameter, the first candidate interface as a selected interface to present the content item;
providing, by the interface management component of the data processing system, the first action data structure to the client computing device for rendering as audio output from the client computing device; and
transmitting, by the interface management component of the data processing system, the content item for rendering at the selected interface.

### Claim 39 (depends on 38)

The method of claim 38, comprising:
paring, by the natural language processor component, the input audio signal to identify a trigger keyword corresponding to the request; and
selecting, by the content selector component and based on the trigger keyword, the content item.

### Claim 40 (depends on 38)

The method of claim 38, wherein first candidate interface is an interface of a second client device.