# US 2022/0284370 A1 · Automatically Learning Process Characteristics for Model Optimization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0284370 A1
- Document ID: US20220284370A1
- Status: Published application
- Kind: A1
- Application number: 17/194,928
- Filed: 2021-03-08
- Published: 2022-09-08
- Earliest claimed: 2021-03-08
- Assignees: Aible Inc.
- Applicants: Aible Inc.
- Inventors: Arijit Sengupta; Jonathan Wray
- Web page: https://patentgrep.com/patent/US20220284370A1

## CPC

- G06Q 10/06375: {Prediction of business process outcome or impact based on a proposed change}
- G06N 20/00: Machine learning
- G06Q 10/06393: {Score-carding, benchmarking or key performance indicator [KPI] analysis}
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Data characterizing inputs to a prediction process that classifies events, an output of the prediction process, and feedback data characterizing a performance of the outcome is monitored. A resource capacity affecting the outcome of the prediction process, and/or a cost-benefit affecting the outcome of the prediction process is determined from the monitoring. The determined resource capacity and/or the determined cost-benefit is provided. Related apparatus, systems, techniques and articles are also described.

## Claims

### Claim 1

A method comprising:
monitoring user activity related to compliance with a predictive model that classifies an event as belonging to a first class of at least two classes, the monitoring including receiving, for each of a plurality of events classified by the predictive model, whether a user took action associated with the first class, a capacity of the user, and a cost-benefit associated with the first class;
determining, based on the received data, an updated capacity and/or an updated cost-benefit; and
providing the updated capacity and/or the updated cost-benefit.

### Claim 2 (depends on 1)

The method of claim 1, wherein the monitoring includes receiving user input characterizing the capacity and/or the cost-benefit; and wherein the determining the updated capacity and/or the updated cost-benefit includes increasing or decreasing the capacity and/or the cost-benefit.

### Claim 3 (depends on 1)

The method of claim 1, wherein the monitoring includes monitoring a field within a dataset for modification that indicates specific action by the user.

### Claim 4 (depends on 1)

The method of claim 1, wherein the determining the updated capacity and/or updated cost-benefit includes determining that the user treats more events as belonging to the first class than the predictive model classifies into the first class.

### Claim 5 (depends on 4)

The method of claim 4, wherein the updated capacity is determined to be greater than the capacity.

### Claim 6 (depends on 4)

The method of claim 4, wherein the updated cost-benefit is determined to include a greater benefit of true positive and/or a lower cost of false positive.

### Claim 7 (depends on 1)

The method of claim 1, wherein the determining the updated capacity and/or updated cost-benefit includes determining that the user treats fewer events as belonging to the first class than the predictive model classifies into the first class.

### Claim 8 (depends on 7)

The method of claim 7, wherein the updated capacity is determined to be less than the capacity.

### Claim 9 (depends on 7)

The method of claim 7, wherein the updated cost-benefit is determined to include a greater cost of false positive and/or a lower benefit of true positive.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
determining that the user treats events classified into a second class by the predictive model as belonging to the first class;
determining a performance of the predictive model;
determining a performance of the user;
determining a new model in response to the performance of the user exceeding the performance of the model.

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
receiving data characterizing performance of a plurality of models associated with a plurality of users, each of the plurality of models associated with a respective user from the plurality of users;
receiving data characterizing, for each of the plurality of users, a respective compliance of the respective user;
determining a differential pattern of performance and compliance across the plurality of models and the plurality of users; and
determining, based on the differential pattern of performance and compliance, a desegregated model performance and a desegregated user compliance.

### Claim 12 (depends on 11)

The method of claim 11, wherein the predictive model forms part of a set of models trained according to respective capacity levels.

### Claim 13 (depends on 1)

The method of claim 1, further comprising:
selecting, in response to determining the updated capacity, a new model from a set of models according to the updated capacity.

### Claim 14 (depends on 1)

The method of claim 1, wherein the event includes a sales opportunity and the first class indicates that the sales opportunity should be pursued.

### Claim 15 (depends on 1)

The method of claim 1, wherein the capacity characterizes a number of events the user processes within a given period of time.

### Claim 16 (depends on 1)

The method of claim 1, wherein the cost-benefit characterizes an impact of treating the event as belonging to the first class, the impact characterized by a cost of a false positive, a cost of a false negative, a benefit of a true positive, and a benefit of a true negative.

### Claim 17

A system comprising:
at least one data processor; and
memory storing instructions which, when executed by the at least one data processor, causes the data processor to perform operations comprising:
monitoring user activity related to compliance with a predictive model that classifies an event as belonging to a first class of at least two classes, the monitoring including receiving, for each of a plurality of events classified by the predictive model, whether a user took action associated with the first class, a capacity of the user, and a cost-benefit associated with the first class;
determining, based on the received data, an updated capacity and/or an updated cost-benefit; and
providing the updated capacity and/or the updated cost-benefit.

### Claim 18 (depends on 17)

The system of claim 17, wherein the monitoring includes receiving user input characterizing the capacity and/or the cost-benefit; and wherein the determining the updated capacity and/or the updated cost-benefit includes increasing or decreasing the capacity and/or the cost-benefit.

### Claim 19 (depends on 17)

The system of claim 17, wherein the monitoring includes monitoring a field within a dataset for modification that indicates specific action by the user.

### Claim 20 (depends on 17)

The system of claim 17, wherein the determining the updated capacity and/or updated cost-benefit includes determining that the user treats more events as belonging to the first class than the predictive model classifies into the first class.

### Claim 21 (depends on 20)

The system of claim 20, wherein the updated capacity is determined to be greater than the capacity.

### Claim 22 (depends on 20)

The system of claim 20, wherein the updated cost-benefit is determined to include a greater benefit of true positive and/or a lower cost of false positive.

### Claim 23 (depends on 17)

The system of claim 17, wherein the determining the updated capacity and/or updated cost-benefit includes determining that the user treats fewer events as belonging to the first class than the predictive model classifies into the first class.

### Claim 24 (depends on 23)

The system of claim 23, wherein the updated capacity is determined to be less than the capacity.

### Claim 25 (depends on 23)

The system of claim 23, wherein the updated cost-benefit is determined to include a greater cost of false positive and/or a lower benefit of true positive.

### Claim 26 (depends on 17)

The system of claim 17, the operations further comprising:
determining that the user treats events classified into a second class by the predictive model as belonging to the first class;
determining a performance of the predictive model;
determining a performance of the user;
determining a new model in response to the performance of the user exceeding the performance of the model.

### Claim 27 (depends on 26)

The system of claim 26, the operations further comprising:
receiving data characterizing performance of a plurality of models associated with a plurality of users, each of the plurality of models associated with a respective user from the plurality of users;
receiving data characterizing, for each of the plurality of users, a respective compliance of the respective user;
determining a differential pattern of performance and compliance across the plurality of models and the plurality of users; and
determining, based on the differential pattern of performance and compliance, a desegregated model performance and a desegregated user compliance.

### Claim 28 (depends on 27)

The system of claim 27, wherein the predictive model forms part of a set of models trained according to respective capacity levels.

### Claim 29 (depends on 17)

The system of claim 17, the operations further comprising:
selecting, in response to determining the updated capacity, a new model from a set of models according to the updated capacity.

### Claim 30 (depends on 17)

The method of claim 17, wherein the cost-benefit characterizes an impact of treating the event as belonging to the first class, the impact characterized by a cost of a false positive, a cost of a false negative, a benefit of a true positive, and a benefit of a true negative.
