# US 10,504,029 B2 · Personalized predictive models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,504,029 B2
- Document ID: US10504029B2
- Status: Granted
- Kind: B2
- Application number: 14/755,253
- Filed: 2015-06-30
- Published: 2017-01-05
- Granted: 2019-12-10
- Earliest claimed: 2015-06-30
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC.
- Inventors: James Edelen; Jian Li; John Fitzgerald Bronskill; John P. Guiver; Kashif Dastgir; Saravanakumar Rajmohan; Artyom Sadovsky
- Web page: https://patentgrep.com/patent/US10504029B2

## CPC

- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 5/02: Knowledge representation; Symbolic representation
- G06N 20/00: Machine learning
- G06Q 10/107: Computer-aided management of electronic mailing [e-mailing]
- G06Q 30/02: Marketing; Price estimation or determination; Fundraising
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Generating and utilizing personalized predictive models are provided. When an electronic input is received, a generic predictive model is used to predict a user response to the input. After a prescribed period of time, an analysis is performed to determine the user's actual response to the input, as well as, the user's actual responses to other inputs of the same type. Training is performed on the generic predictive model to generate a new and personalized predictive model based on the user's actual responses to the analyzed inputs. The personalized predictive model is then utilized for predicting user response to future inputs of the same type. At a prescribed frequency, the generated personalized predictive model is updated by analyzing actual user responses to predictions provided by the personalized predictive model.

## Claims

### Claim 1

A system for generating a personalized predictive model; comprising:
a user interface;
a processor in communication with the user interface; and
a memory including instructions, which when executed by the processor, cause the system to operate:
an email application operative to:
receive an email, wherein the email comprises two or more features characterizing the email;
present the email to the user interface for a user to make a user decision;
a generic predictive modeling module operative to:
generate a generic predictive model for predicting the user decision on the received email;
utilize a generic precision/recall curve for obtaining a user decision threshold for determining a percentage of received emails to be decided according to a prescribed decision by the user, wherein the generic precision/recall curve is generated from at least another user's response to a same or similar email, and wherein which generic precision/recall curve is utilized may be based, at least in part, on the two or more features;
a usage monitoring module operative to compare an actual user decision on the received email with the predicted user decision for determining an accuracy of the predicted user decision;
a personalized predictive model training module operative to update one or more features of the generic predictive model for personalizing the generic predictive model as a personalized predictive model for a user of the received email;
a personalized predictive modeling module operative to:
generate the personalized predictive model based on the one or more updated features for predicting a decision of the user on one or more future emails:
update the personalized predictive model periodically based on user inputs during to improve accuracy; and
wherein the email application applies the personalized predictive model to move automatically an email message to a junk folder without input from the user.

### Claim 2 (depends on 1)

The system of claim 1, the usage monitoring module being further operative to compare a plurality of actual user decisions with a plurality of corresponding predicted user decisions, the actual user decisions and the corresponding predicted user decisions accumulated over a prescribed period of time.

### Claim 3 (depends on 1)

The system of claim 1, the personalized predictive model training module being further operative to compute an updated precision/recall curve that is personalized to the receiving user based on the plurality of actual user decisions and corresponding predicted user decisions accumulated over the prescribed period of time.

### Claim 4 (depends on 3)

The system of claim 3, the personalized predictive model training module being further operative to determine a plurality of user decision thresholds that will improve accuracy of future predicted user decisions on future emails.

### Claim 5 (depends on 4)

The system of claim 4, the personalized predictive model training module being further operative to compute an optimum user decision threshold for use in the personalized predictive model for the future predicted user decisions.

### Claim 6 (depends on 5)

The system of claim 5, the personalized predictive module being further operative to generate an updated personalized predictive model based on the optimum user decision threshold for use in predicting future user decisions on future emails.

### Claim 7 (depends on 6)

The system of claim 6, the personalized prediction modeling module being further operative to update the updated personalized predictive model based on an updated optimum user decision threshold generated in response to an analysis of predicted future user decisions compared with actual user decisions on the future emails.

### Claim 8 (depends on 1)

The system of claim 1, wherein the system for generating a personalized predictive model is operative to communicate the personalized predictive model to the email application for causing the email application to act on one or more future emails received by a user of the email application.

### Claim 9 (depends on 8)

The system of claim 8, wherein one or more decisions by the user of the email application on the one or more future emails are monitored by the usage monitoring module for determining an accuracy of predictions made by the personalized predictive model for responses to the one or more future emails by the user of the email application.

### Claim 10 (depends on 9)

The system of claim 9, wherein the personalized predictive modeling module is further operative to update the generated personalized predictive model based on the one or more decisions by the user of the email application on the one or more future emails.

### Claim 11

A method for improving the functionality of a computing device on which is operated a email application with functionality requiring user action on received emails, comprising:
receiving, by the email application executing on the computing device, an email, wherein the email comprises two or more features characterizing the email;
presenting, on a user interface of the computing device, the email to a user for a user decision;
predicting the user decision on the received email via a generic predictive model operated by the computing device, wherein the generic predictive model utilizes a generic precision/recall curve for obtaining a user decision threshold for determining a percentage of received emails to be decided according to a prescribed decision by the user, wherein the generic precision/recall curve is generated from at least another user's response to a same or similar email, and wherein which generic precision/recall curve is utilized may be based, at least in part, on the two or more features;
comparing an actual user decision on the received email with the predicted user decision for determining an accuracy of the predicted user decision;
updating one or more features of the generic predictive model for personalizing the generic predictive model as a personalized predictive model for a user of the received email;
performing a computational loop for a precision recall curve associated each of two or more potential thresholds to determine best threshold value for the personalized predictive model;
generating the personalized predictive model based on the one or more updated features for predicting a decision of the user on one or more future emails; and
applying the personalized predictive model to move automatically an email message to a junk folder without input from the user.

### Claim 12 (depends on 11)

The method of claim 11, further comprising comparing a plurality of actual user decisions with a plurality of corresponding predicted user decisions, the actual user decisions and the corresponding predicted user decisions accumulated over a prescribed period of time.

### Claim 13 (depends on 11)

The method of claim 11, further comprising computing an updated precision/recall curve that is personalized to the receiving user based on the plurality of actual user decisions and corresponding predicted user decisions accumulated over the prescribed period of time.

### Claim 14 (depends on 13)

The method of claim 13, further comprising determining a plurality of user decision thresholds that will improve accuracy of future predicted user decision on future emails.

### Claim 15 (depends on 14)

The method of claim 14, further comprising computing an optimum user decision threshold for use in the personalized predictive model for the future predicted user decisions.

### Claim 16 (depends on 15)

The method of claim 15, further comprising generating an updated personalized predictive model based on the optimum user decision threshold for use in predicting future user decisions on future emails.

### Claim 17

A method for predicting a user's decisions on received emails; comprising:
receiving via a email application an email requiring a decision by a receiving user, wherein the email comprises two or more features characterizing the email;
presenting, on a user interface of the computing device, the email to a user for the user decision;
using a generic predictive model, predicting the user decision on a received email, wherein the generic predictive model utilizes a generic precision/recall curve for obtaining a user decision threshold for determining a percentage of received emails to be decided according to a prescribed decision by the user, wherein the generic precision/recall curve is generated from at least another user's response to a same or similar email, and wherein which generic precision/recall curve is utilized may be based, at least in part, on the two or more features;
performing a computational loop for a precision recall curve associated with each of two or more potential thresholds to determine best threshold value for the personalized predictive model;
comparing an actual user decision on the received email with the predicted user decision for determining an accuracy of the predicted user decision;
generating a personalized predictive model by updating one or more features of the generic predictive model for personalizing the generic predictive model as a personalized predictive model for a user of the received email;
using the personalized predictive model for predicting a user decision for each of a plurality of future emails;
based on the prediction, moving automatically an email message to a junk folder without input from the user
after a prescribed duration of time, comparing an actual user decision for each of the plurality of future emails with corresponding predicted user decisions for each of the plurality of future emails; and
updating the personalized predictive model for more accurately predicting a user decision for each of a plurality of additional future emails.

### Claim 18 (depends on 17)

The method of claim 17, wherein updating the personalized predictive model includes:
computing an updated precision/recall curve that is personalized to the receiving user based on the plurality of actual user decisions and corresponding predicted user decisions accumulated over the prescribed duration of time;
determining a plurality of user decision thresholds that will improve accuracy of future predicted user decision on future emails;
computing an optimum user decision threshold for use in the personalized predictive model for the future predicted user decisions; and
utilizing the optimum user decision threshold for predicting future user decisions on the plurality of additional future emails.