# US 11,443,200 B2 · Automated decision analysis by model operational characteristic curves

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,443,200 B2
- Document ID: US11443200B2
- Status: Granted
- Kind: B2
- Application number: 15/780,453
- Filed: 2018-03-06
- Published: 2020-06-04
- Granted: 2022-09-13
- Earliest claimed: 2018-03-06
- Assignees: Visa International Service Association
- Applicants: Visa International Service Association
- Inventors: Hung-Tzaw Hu; Benjamin Scott Boding; Ge Wen; Haochuan Zhou
- Web page: https://patentgrep.com/patent/US11443200B2

## CPC

- G06N 5/04: Inference or reasoning models
- G06Q 30/0609: {Qualifying participants for shopping transactions (payment transaction verification G06Q20/401)}
- G06N 20/00: Machine learning
- G06Q 10/04: Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" (market predictions or forecasting for commercial activities G06Q30/0202)
- G06Q 20/40: Authorisation, e.g. identification of payer or payee, verification of customer or shop credentials; Review and approval of payers, e.g. check credit lines or negative lists
- G06Q 20/4016: {involving fraud or risk level assessment in transaction processing}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Described are a system, method, and computer program product for optimizing a predictive condition classification model and automatically enacting reactive measures based thereon. The method includes receiving event data representative of a plurality of events. The method also includes receiving the predictive condition classification model configured to categorize each event as satisfying a condition or not. The predictive condition classification model is configured to order the plurality of events by likelihood of satisfying the condition. The method includes generating a performance evaluation dataset and plotting data configured to cause a visual display to represent at least two model performance metrics of the performance evaluation dataset on a same output plot. The method includes automatically rejecting a top percent of the plurality of events for suspected satisfaction of the condition, determined at least partially from a customized rejection algorithm or a preset rejection algorithm.

## Claims

### Claim 1

A computer-implemented method for optimizing a predictive fraud detection model and automatically enacting reactive measures based thereon, the method comprising:
receiving, with at least one processor, transaction data representative of a plurality of transaction requests between at least one financial device holder and at least one merchant in a first time period, the transaction data comprising for each transaction request of the plurality of transaction requests at least one of the following parameters: transaction date, transaction time, transaction value, merchant type, transaction type, merchant location, or any combination thereof;
receiving, with at least one processor from a user interface, an identification of the predictive fraud detection model to be evaluated;
in response to receiving the identification, receiving, with at least one processor, the predictive fraud detection model to be evaluated, the predictive fraud detection model configured to categorize each transaction request as fraudulent or legitimate based at least partially on the parameters of the transaction request, the predictive fraud detection model further configured to order the plurality of transaction requests, relative to each other transaction request, from most likely fraudulent to least likely fraudulent;
generating, with at least one processor, a performance evaluation dataset comprising an output of the predictive fraud detection model given an input of the transaction data;
generating, with at least one processor, plotting data configured to cause a visual display of the user interface to represent at least two model performance metrics of the performance evaluation dataset on a same output plot, the same output plot having an x-axis of percentile threshold for transaction rejection and a y-axis of percent of metric value of the at least two model performance metrics, the at least two model performance metrics calculated from the performance evaluation dataset, and the at least two model performance metrics being a false positive ratio and at least one of the following: detection rate, maximum detectable rate, minimum false positive ratio, model performance score, random model false positive ratio, random model detection rate, or any combination thereof;
transmitting, with at least one processor, the plotting data to the user interface, wherein the plotting data is further configured to, in response to input in the user interface comprising a specification of a quantitative value of percentile threshold for transaction rejection, generate a corresponding metric value of each of the at least two model performance metrics in the same output plot;
receiving, with at least one processor from the user interface, a user input comprising the specification of the quantitative value of percentile threshold for transaction rejection; and
in response to receiving the user input, automatically rejecting, with at least one processor, a top percent of the plurality of transaction requests for suspected fraudulent transaction activity, the top percent determined at least partially from a customized rejection algorithm based on the user input.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, the method further comprising, in response to receiving a metric input in the user interface of a first value of a first selected model performance metric, generating and displaying, with at least one processor, a corresponding value of a second selected model performance metric, wherein the plotting data is further configured to cause the visual display to represent all of the following model performance metrics on the same output plot: detection rate, false positive ratio, maximum detectable rate, minimum false positive ratio, model performance score, random model false positive ratio, and random model detection rate.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising:
receiving, with at least one processor, an extrinsic evaluation of a legitimacy of each transaction request of the top percent of the plurality of transaction requests, the extrinsic evaluation comprising a true positive rate and a false positive rate of the performance evaluation dataset;
adjusting, with at least one processor, the predictive fraud detection model based on the extrinsic evaluation;
receiving, with at least one processor, a second set of transaction data representative of a plurality of new transaction requests between at least one financial device holder and at least one merchant in a second time period;
re-generating, with at least one processor, the performance evaluation dataset;
re-generating, with at least one processor, the plotting data; and,
automatically rejecting, with at least one processor and based on the adjusted predictive fraud detection model, a top percent of the plurality of new transaction requests for suspected fraudulent transaction activity.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, further comprising receiving, with at least one processor, at least one additional predictive fraud detection model to be evaluated; wherein:
the performance evaluation dataset further comprises an output of the at least one additional predictive fraud detection model to be evaluated; and,
the plotting data is further configured to cause the visual display to represent the predictive fraud detection model comparatively to the at least one additional predictive fraud detection model.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, further comprising:
receiving, with at least one processor, a selected model from the user interface, the selected model chosen from a set comprising the predictive fraud detection model and the at least one additional predictive fraud detection model,
wherein the automatic rejection of the top percent of the plurality of transaction requests is based on the selected model, which is configured to order the plurality of transaction requests from most likely fraudulent to least likely fraudulent.

### Claim 6 (depends on 4)

The computer-implemented method of claim 4, wherein the user input further comprises an optimization input comprising at least one of the following parameters: detect rate requirement; false positive ratio tolerance; rejection rate tolerance; review rate capacity; or any combination thereof; and wherein the top percent is further determined from the optimization input.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the user input further comprises an input cost tolerance, and wherein the customized rejection algorithm is further based on an optimization input determined from the input cost tolerance and an associated cost incurred per rejected transaction request.

### Claim 8

A system for optimizing a predictive fraud detection model and automatically enacting reactive measures based thereon, the system comprising at least one server computer including at least one processor, the at least one server computer programmed and/or configured to:
receive transaction data representative of a plurality of transaction requests between at least one financial device holder and at least one merchant in a first time period, the transaction data comprising for each transaction request of the plurality of transaction requests at least one of the following parameters: transaction date, transaction time, transaction value, merchant type, transaction type, merchant location, or any combination thereof;
receive, from a user interface, an identification of the predictive fraud detection model to be evaluated;
in response to receiving the identification, receive the predictive fraud detection model to be evaluated, the predictive fraud detection model configured to categorize each transaction request as fraudulent or legitimate based at least partially on the parameters of the transaction request, the predictive fraud detection model further configured to order the plurality of transaction requests, relative to each other transaction request, from most likely fraudulent to least likely fraudulent;
generate a performance evaluation dataset comprising an output of the predictive fraud detection model given an input of the transaction data;
generate plotting data configured to cause a visual display of the user interface to represent at least two model performance metrics of the performance evaluation dataset on a same output plot, the same output plot having an x-axis of percentile threshold for transaction rejection and a y-axis of percent of metric value of the at least two model performance metrics, the at least two model performance metrics calculated from the performance evaluation dataset, and the at least two model performance metrics being a false positive ratio and at least one of the following: detection rate, maximum detectable rate, minimum false positive ratio, model performance score, random model false positive ratio, random model detection rate, or any combination thereof;
transmit the plotting data to the user interface, wherein the plotting data is further configured to, in response to input in the user interface comprising a specification of a quantitative value of percentile threshold for transaction rejection, generate a corresponding metric value of each of the at least two model performance metrics in the same output plot;
receive a user input comprising the specification of the quantitative value of percentile threshold for transaction rejection; and
in response to receiving the user input, automatically reject a top percent of the plurality of transaction requests for suspected fraudulent transaction activity, the top percent determined at least partially from a customized rejection algorithm based on the user input.

### Claim 9 (depends on 8)

The system of claim 8, the at least one server computer further programmed and/or configured to, in response to receiving a metric input in the user interface of a first value of a first selected model performance metric, generate and display a corresponding value of a second selected model performance metric, wherein the plotting data is further configured to cause the visual display to represent all of the following model performance metrics on the same output plot: detection rate, false positive ratio, maximum detectable rate, minimum false positive ratio, model performance score, random model false positive ratio, and random model detection rate.

### Claim 10 (depends on 8)

The system of claim 8, the at least one server computer further programmed and/or configured to:
receive an extrinsic evaluation of a legitimacy of each transaction request of the top percent of the plurality of transaction requests, the extrinsic evaluation comprising a true positive rate and a false positive rate of the performance evaluation dataset;
adjust the predictive fraud detection model based on the extrinsic evaluation;
receive a second set of transaction data representative of a plurality of new transaction requests between at least one financial device holder and at least one merchant in a second time period;
re-generate the performance evaluation dataset;
re-generate the plotting data; and,
automatically reject, based on the adjusted predictive fraud detection model, a top percent of the plurality of new transaction requests for suspected fraudulent transaction activity.

### Claim 11 (depends on 8)

The system of claim 8, the at least one server computer further programmed and/or configured to receive at least one additional predictive fraud detection model to be evaluated; wherein:
the performance evaluation dataset further comprises an output of the at least one additional predictive fraud detection model to be evaluated; and,
the plotting data is further configured to cause the visual display to represent the predictive fraud detection model comparatively to the at least one additional predictive fraud detection model.

### Claim 12 (depends on 11)

The system of claim 11, the at least one server computer further programmed and/or configured to:
receive a selected model from the user interface, the selected model chosen from a set comprising the predictive fraud detection model and the at least one additional predictive fraud detection model,
wherein the automatic rejection of the top percent of the plurality of transaction requests is based on the selected model, which is configured to order the plurality of transaction requests from most likely fraudulent to least likely fraudulent.

### Claim 13 (depends on 11)

The system of claim 11, wherein the user input further comprises an optimization input comprising at least one of the following parameters: detect rate requirement; false positive ratio tolerance; rejection rate tolerance; review rate capacity; or any combination thereof; and wherein the top percent is further determined from the optimization input.

### Claim 14 (depends on 8)

The system of claim 8, wherein the user input further comprises an input cost tolerance, and wherein the customized rejection algorithm is further based on an optimization input determined from the input cost tolerance and an associated cost incurred per rejected transaction request.

### Claim 15

A computer program product for optimizing a predictive fraud detection model and automatically enacting reactive measures based thereon, the computer program product comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to:
receive transaction data representative of a plurality of transaction requests between at least one financial device holder and at least one merchant in a first time period, the transaction data comprising for each transaction request of the plurality of transaction requests at least one of the following parameters: transaction date, transaction time, transaction value, merchant type, transaction type, merchant location, or any combination thereof;
receive, from a user interface, an identification of the predictive fraud detection model to be evaluated;
in response to receiving the identification, receive the predictive fraud detection model to be evaluated, the predictive fraud detection model configured to categorize each transaction request as fraudulent or legitimate based at least partially on the parameters of the transaction request, the predictive fraud detection model further configured to order the plurality of transaction requests, relative to each other transaction request, from most likely fraudulent to least likely fraudulent;
generate a performance evaluation dataset comprising an output of the predictive fraud detection model given an input of the transaction data;
generate plotting data configured to cause a visual display of the user interface to represent at least two model performance metrics of the performance evaluation dataset on a same output plot, the same output plot having an x-axis of percentile threshold for transaction rejection and a y-axis of percent of metric value of the at least two model performance metrics, the at least two model performance metrics calculated from the performance evaluation dataset, and the at least two model performance metrics being a false positive ratio and at least one of the following: detection rate, maximum detectable rate, minimum false positive ratio, model performance score, random model false positive ratio, random model detection rate, or any combination thereof; transmit the plotting data to the user interface, wherein the plotting data is further configured to, in response to input in the user interface comprising a specification of a quantitative value of percentile threshold for transaction rejection, generate a corresponding metric value of each of the at least two model performance metrics in the same output plot;
receive a user input comprising the specification of the quantitative value of percentile threshold for transaction rejection; and
in response to receiving the user input, automatically reject a top percent of the plurality of transaction requests for suspected fraudulent transaction activity, the top percent determined at least partially from a customized rejection algorithm based on the user input.

### Claim 16 (depends on 15)

The computer program product of claim 15, wherein the program instructions further cause the at least one processor to, in response to receiving a metric input in the user interface of a first value of a first selected model performance metric, generate and display a corresponding value of a second selected model performance metric, wherein the plotting data is further configured to cause the visual display to represent all of the following model performance metrics on the same output plot: detection rate, false positive ratio, maximum detectable rate, minimum false positive ratio, model performance score, random model false positive ratio, and random model detection rate.

### Claim 17 (depends on 15)

The computer program product of claim 15, wherein the program instructions further cause the at least one processor to:
receive an extrinsic evaluation of a legitimacy of each transaction request of the top percent of the plurality of transaction requests, the extrinsic evaluation comprising a true positive rate and a false positive rate of the performance evaluation dataset;
adjust the predictive fraud detection model based on the extrinsic evaluation;
receive a second set of transaction data representative of a plurality of new transaction requests between at least one financial device holder and at least one merchant in a second time period;
re-generate the performance evaluation dataset;
re-generate the plotting data; and,
automatically reject, based on the adjusted predictive fraud detection model, a top percent of the plurality of new transaction requests for suspected fraudulent transaction activity.

### Claim 18 (depends on 15)

The computer program product of claim 15, wherein the program instructions further cause the at least one processor to receive at least one additional predictive fraud detection model to be evaluated; wherein:
the performance evaluation dataset further comprises an output of the at least one additional predictive fraud detection model to be evaluated; and,
the plotting data is further configured to cause the visual display to represent the predictive fraud detection model comparatively to the at least one additional predictive fraud detection model.

### Claim 19 (depends on 18)

The computer program product of claim 18, wherein the program instructions further cause the at least one processor to:
receive a selected model from the user interface, the selected model chosen from a set comprising the predictive fraud detection model and the at least one additional predictive fraud detection model,
wherein the automatic rejection of the top percent of the plurality of transaction requests is based on the selected model, which is configured to order the plurality of transaction requests from most likely fraudulent to least likely fraudulent.

### Claim 20 (depends on 18)

The computer program product of claim 18, wherein the user input further comprises an optimization input comprising at least one of the following parameters: detect rate requirement; false positive ratio tolerance; rejection rate tolerance; review rate capacity; or any combination thereof; and wherein the top percent is further determined from the optimization input.

### Claim 21 (depends on 15)

The computer program product of claim 15, wherein the user input further comprises an input cost tolerance, and wherein the customized rejection algorithm is further based on an optimization input determined from the input cost tolerance and an associated cost incurred per rejected transaction request.