# US 11,113,604 B2 · Training and/or utilizing an interaction prediction model to determine when to interact, and/or prompt for interaction, with an application on the basis of an electronic communication

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,113,604 B2
- Document ID: US11113604B2
- Status: Granted
- Kind: B2
- Application number: 15/815,478
- Filed: 2017-11-16
- Published: 2019-05-09
- Granted: 2021-09-07
- Earliest claimed: 2017-11-06
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Sebastian Millius; Tom Hume; Matthew Sharifi
- Web page: https://patentgrep.com/patent/US11113604B2

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06F 40/216: using statistical methods
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F 40/30: Semantic analysis
- G06F 40/35: Discourse or dialogue representation
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06Q 10/107: Computer-aided management of electronic mailing [e-mailing]
- H04L 51/046: Interoperability with other network applications or services
- H04L 51/234: for tracking messages
- H04M 1/72403: with means for local support of applications that increase the functionality
- H04M 1/72484: wherein functions are triggered by incoming communication events
- G06F 40/169: Annotation, e.g. comment data or footnotes
- H04M 1/72436: for text messaging, e.g. short messaging services [SMS] or e-mails
- H04M 1/72451: according to schedules, e.g. using calendar applications
- H04M 1/72454: according to context-related or environment-related conditions
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)

## Abstract

Training and/or utilizing an interaction prediction model to generate a predicted interaction value that indicates a likelihood of interaction with a corresponding application on the basis of an electronic communication. The application can be in addition to any electronic communication application that is utilized in formulating the electronic communication and/or that is utilized in rendering the electronic communication. The predicted interaction value can be generated based on processing, utilizing the interaction prediction model, of features of the electronic communication and/or of other features. The predicted interaction value can be utilized to determine whether to perform further action(s) that interact with, and/or enable efficient interaction with, the application on the basis of the electronic communication.

## Claims

### Claim 1

A method, comprising:
identifying, by a client device, an electronic communication,
the electronic communication including content that comprises natural language content, and
the electronic communication being received at the client device via a network interface of the client device, or being formulated at the client device via at least one user interface input device of the client device;
applying, by the client device, as at least part of first input to a first portion of a trained machine learning model that is stored locally at the client device:
at least part of the natural language content of the electronic communication;
determining, by the client device, one or more additional input features;
applying, by the client device, as at least part of second input to a second portion of the trained machine learning model stored locally at the client device:
at least one of the one or more additional input features;
processing, by the client device, the first and second input using the trained machine learning model to generate a predicted interaction value,
the predicted interaction value indicating a likelihood of interaction with a second application accessible to the client device, the interaction with the second application being on the basis of the electronic communication,
wherein the second application is in addition to an electronic communication application via which the electronic communication is formulated, or via which the electronic communication is rendered after being received;
determining, by the client device and based on the predicted interaction value, whether to present a selectable interface element via a display of the client device,
wherein the selectable interface element, when selected via user interface input at the client device, causes interaction with the second application to provide at least part of the content of the electronic communication to the second application;
determining, by the client device, whether any interaction, with the second application and on the basis of the electronic communication, occurs in response to selection of the selectable interface element or in response to other user interface input provided at the client device;
updating, by the client device, trained parameters of at least part of the trained machine learning model based on whether any interaction occurred with the second application on the basis of the electronic communication, wherein updating the trained parameters comprises:
determining a gradient based on whether any interaction occurred with the second application on the basis of the electronic communication, and based on the predicted interaction value, and
backpropagating the gradient over at least a portion of the trained machine learning model; and
using, by the client device, the trained machine learning model with the updated trained parameters in determining whether to present an additional selectable interface element for an additional electronic communication received or formulated at the client device.

### Claim 2 (depends on 1)

The method of claim 1, wherein processing the first input using the trained machine learning model comprises processing the at least part of the first input using the first portion to generate first portion intermediary output, and processing the first portion intermediary output using the second portion, and wherein updating the trained parameters of the at least part of the trained machine learning model comprises:
updating the second portion of the trained machine learning model without updating the first portion.

### Claim 3 (depends on 1)

The method of claim 1, wherein the one or more additional features are determined based on at least one of:
sensor data from one or more sensors of the client device; and
application data related to the electronic communication application via which the electronic communication is formulated, or via which the electronic communication is rendered in response to being received.

### Claim 4 (depends on 3)

The method of claim 3, wherein the one or more additional features are determined based on the sensor data from the one or more sensors of the client device, and wherein the one or more sensors include at least one of a positional sensor, an accelerometer, and an inertial measurement unit.

### Claim 5 (depends on 4)

The method of claim 4, wherein the one or more additional features include a predicted status, of a user of the client device, that is predicted based on the sensor data.

### Claim 6 (depends on 1)

The method of claim 1, wherein the one or more additional features are determined based on one or more current temporal indications.

### Claim 7 (depends on 1)

The method of claim 1, wherein processing the first and second input using the trained machine learning model is further to generate an additional predicted interaction value, the additional predicted interaction value indicating a likelihood of interaction with an additional application, accessible to the client device, on the basis of the electronic communication.

### Claim 8 (depends on 1)

The method of claim 1, wherein determining whether any interaction with the second application on the basis of the electronic communication occurs, is in response to the other user interface input provided at the client device, and wherein the other user interface input comprises pasting or typing at least some of the natural language input into the second application.

### Claim 9 (depends on 1)

The method of claim 1, wherein the electronic communication application is a messaging application, and wherein the second application is a calendar application, a notes application, or a reminder application.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
generating, based on processing of tokens of the natural language content using a separate trained machine learning model, the at least part of the content of the electronic communication to provide to the second application, wherein the at least part of the content is a subset of the content; and
based on the at least part of the content being generated using the separate trained machine learning model:
causing selection of the selectable interface element to cause the interaction with the second application to provide the at least part of the content to the application.

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
incorporating at least a segment of the at least part of the content into the selectable interface element.

### Claim 12 (depends on 10)

The method of claim 10, wherein causing the interaction with the second application to provide the at least part of the content to the second application comprises causing the at least part of the content to be communicated to the second application utilizing an application programming interface of the second application.

### Claim 13 (depends on 1)

The method of claim 1, wherein the electronic communication is received at the client device via the network interface, and wherein the electronic communication application is a messaging application.

### Claim 14 (depends on 1)

The method of claim 1, wherein the electronic communication is formulated at the client device via at least one user interface input device of the client device, and wherein the at least part of the natural language content is received on a token-by-token basis via a keyboard application that renders an electronic keyboard via which the electronic communication is formulated.

### Claim 15

A method, comprising:
identifying, by a client device, an electronic communication,
the electronic communication including content that comprises natural language content, and
the electronic communication being received at the client device via a network interface of the client device, or being formulated at the client device via at least one user interface input device of the client device;
determining, by the client device, whether any interaction occurs with a second application on the basis of the electronic communication,
wherein the second application is in addition to an electronic communication application via which the electronic communication is formulated, or via which the electronic communication is rendered after being received;
updating, by the client device, trained parameters of at least part of a trained machine learning model based on whether any interaction occurred with the second application on the basis of the electronic communication, wherein updating the trained parameters comprises:
determining a gradient based on whether any interaction occurred with the second application on the basis of the electronic communication, and based on a prediction value generated based on processing of features, of the electronic communication, using the trained machine learning model, and
backpropagating the gradient over at least a portion of the trained machine learning model; and
using, by the client device, the trained machine learning model with the updated trained parameters in determining whether to present an additional selectable interface element for an additional electronic communication received or formulated at the client device.

### Claim 16

A method performed by one or more processors of a client device, comprising:
identifying, by a client device, an electronic communication,
the electronic communication including content that comprises natural language content, and
the electronic communication being received at the client device via a network interface of the client device, or being formulated at the client device via at least one user interface input device of the client device;
applying, by the client device, as at least part of input to a trained machine learning model that is stored locally at the client device:
tokens of the natural language content of the electronic communication;
processing, by the client device, the input using the trained machine learning model to generate a predicted interaction value,
the predicted interaction value indicating a likelihood of interaction with a second application, accessible to the client device, on the basis of the electronic communication,
wherein the second application is in addition to an electronic communication application via which the electronic communication is formulated, or via which the electronic communication is rendered after being received;
determining, by the client device based on the predicted interaction value, to present a selectable interface element via a display of the client device;
generating, based on processing of the tokens of the natural language content of the electronic communication using a separate trained machine learning model, at least part of content of the electronic communication to provide to the second application, wherein the at least part of the content is a subset of the content of the electronic communication;
based on the at least part of the content being generated using the separate trained machine learning model:
causing selection of the selectable interface element to cause the interaction with the second application to provide the at least part of the content to the second application; and
in response to selection of the selectable interface element:
interfacing with the second application to provide at least part of the content of the electronic communication to the second application.

### Claim 17 (depends on 16)

The method of claim 16, wherein the trained machine learning model includes a first portion and a second portion, and wherein processing the input using the trained machine learning model comprises processing the tokens using the first portion to generate first portion intermediary output, and processing the first portion intermediary output using the second portion.

### Claim 18 (depends on 17)

The method of claim 17, further comprising:
determining, by the client device, one or more additional features;
applying, by the client device, the one or more additional features to the second portion of the trained machine learning model as an additional part of the input to the trained machine learning model;
wherein processing the input using the trained machine learning model further comprises processing the one or more additional features using the second portion.

### Claim 19 (depends on 16)

The method of claim 16, wherein processing the input using the trained machine learning model is further to generate an additional predicted interaction value, the additional predicted interaction value indicating a likelihood of interaction with an additional application, accessible to the client device, on the basis of the electronic communication, and further comprising:
determining, based on the predicted interaction value and the additional predicted interaction value:
to present the selectable interface element that, when selected, interfaces with the second application, without presenting any selectable interface element that, when selected, interfaces with the additional application to provide any part of the content of the electronic communication to the additional application.

### Claim 20 (depends on 16)

The method of claim 16, wherein generating the at least part of the content of the electronic communication to provide to the second application is contingent on determining that the predicted interaction value satisfies a threshold.