Back to results
US 2026/0278480 A1Published application
System, Method, and Computer Program Product for Feature Similarity-Based Monitoring and Validation of Models
AssigneeVisa International Service Association
InventorsXi Kan, Sheng Wang, Shuo Wang, Pei Yang, Dan Wang
Life of this patent
Estimating the term…
Abstract
Systems, methods, and computer program products calculate a historical feature similarity point distribution associated with a first machine learning model, calculate a real-time feature similarity point distribution associated with a second machine learning model, and automatically provide, based on a comparison of the historical feature similarity point distribution to the real-time feature similarity point distribution, an indication of whether the second machine learning model is aligned with the first machine learning model.
Claims20
A method, comprising:
- obtaining, with at least one processor, a plurality of training samples and a plurality of first machine learning model outputs generated by a first machine learning model based on the plurality of training samples, wherein the plurality of training samples is associated with a plurality of historical payment transactions processed in a transaction processing network;
- providing, with the at least one processor, a plurality of training sample groups based on the plurality of training samples and the plurality of first machine learning model outputs;
- determining, with the at least one processor, a feature similarity point for each pair of training samples in each training sample group of the plurality of training sample groups, wherein the feature similarity point for a pair of training samples is determined based on features of each training sample in the pair of training samples;
- obtaining, with the at least one processor, an inference sample and a second machine learning model output generated by a second machine learning model based on the inference sample, wherein the inference sample is associated with a real-time payment transaction currently being processed in the transaction processing network, wherein the real-time payment transaction comprises a first plurality of transaction parameters, and wherein the second machine learning model output comprises a first authorization decision;
- classifying, with the at least one processor, the inference sample into a first training sample group of a plurality of training sample groups based on the second machine learning model output;
- determining, with the at least one processor, a real-time feature similarity point distribution associated with the first training sample group based on each training sample of the first training sample group, wherein the real-time feature similarity point distribution includes a feature similarity point for each training sample of the first training sample group and the inference sample, and wherein determining the real-time feature similarity point distribution comprises:
- determining a feature similarity point for the inference sample and each training sample of the first training sample group;
- automatically providing an indication of whether the second machine learning model is aligned with the first machine learning model based on a comparison of a historical feature similarity point distribution to the real-time feature similarity point distribution, wherein the historical feature similarity point distribution of each training sample group includes the feature similarity point for each pair of training samples in the first training sample group;
- providing, with the at least one processor, a second plurality of historical payment transactions in response to an indication that the second machine learning model is not aligned with the first machine learning model, wherein transaction data associated with each historical payment transaction of the second plurality of historical payment transactions comprises a second plurality of transaction parameters and an authorization decision, and wherein the second plurality of historical payment transactions comprises payment transactions having an authorization decision different from the first authorization decision;
- determining, with the at least one processor, at least one impact parameter of the first plurality of transaction parameters;
- and
- generating, with the at least one processor, a modified first machine learning model using a loss function that depends on the at least one impact parameter, wherein generating the modified first machine learning model comprises:
- modifying one or more weights or parameters of the first machine learning model using the loss function that depends on the at least one impact parameter until the second plurality of historical payment transactions no longer cause the first machine learning model to generate authorization decisions different from the first authorization decision.
10 more elements
A system, comprising:
- at least one processor programmed or configured to:
- obtain a plurality of training samples and a plurality of first machine learning model outputs generated by a first machine learning model based on the plurality of training samples, wherein the plurality of training samples is associated with a plurality of historical payment transactions processed in a transaction processing network;
- provide a plurality of training sample groups based on the plurality of training samples and the plurality of first machine learning model outputs;
- determine a feature similarity point for each pair of training samples in each training sample group of the plurality of training sample groups, wherein the feature similarity point for a pair of training samples is determined based on features of each training sample in the pair of training samples;
- obtain an inference sample and a second machine learning model output generated by a second machine learning model based on the inference sample, wherein the inference sample is associated with a real-time payment transaction currently being processed in the transaction processing network, wherein the real-time payment transaction comprises a first plurality of transaction parameters, and wherein the second machine learning model output comprises a first authorization decision;
- classify the inference sample into a first training sample group of a plurality of training sample groups based on the second machine learning model output;
- determine a real-time feature similarity point distribution associated with the first training sample group based on each training sample of the first training sample group, wherein the real-time feature similarity point distribution includes a feature similarity point for each training sample of the first training sample group and the inference sample, and wherein, when determining the real-time feature similarity point distribution, the at least one processor is programmed or configured to:
- determine a feature similarity point for the inference sample and each training sample of the first training sample group;
- automatically provide an indication of whether the second machine learning model is aligned with the first machine learning model based on a comparison of a historical feature similarity point distribution to the real-time feature similarity point distribution, wherein the historical feature similarity point distribution of each training sample group includes the feature similarity point for each pair of training samples in the first training sample group;
- provide a second plurality of historical payment transactions in response to an indication that the second machine learning model is not aligned with the first machine learning model, wherein transaction data associated with each historical payment transaction of the second plurality of historical payment transactions comprises a second plurality of transaction parameters and an authorization decision, and wherein the second plurality of historical payment transactions comprises payment transactions having an authorization decision different from the first authorization decision;
- determine at least one impact parameter of the first plurality of transaction parameters;
- and
- generate a modified first machine learning model using a loss function that depends on the at least one impact parameter, wherein, when generating the modified first machine learning model, the at least one processor is programmed or configured to:
- modify one or more weights or parameters of the first machine learning model using the loss function that depends on the at least one impact parameter until the second plurality of historical payment transactions no longer cause the first machine learning model to generate authorization decisions different from the first authorization decision.
11 more elements
A computer program product comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to:
- obtain a plurality of training samples and a plurality of first machine learning model outputs generated by a first machine learning model based on the plurality of training samples, wherein the plurality of training samples is associated with a plurality of historical payment transactions processed in a transaction processing network;
- provide a plurality of training sample groups based on the plurality of training samples and the plurality of first machine learning model outputs;
- determine a feature similarity point for each pair of training samples in each training sample group of the plurality of training sample groups, wherein the feature similarity point for a pair of training samples is determined based on features of each training sample in the pair of training samples;
- obtain an inference sample and a second machine learning model output generated by a second machine learning model based on the inference sample, wherein the inference sample is associated with a real-time payment transaction currently being processed in the transaction processing network, wherein the real-time payment transaction comprises a first plurality of transaction parameters, and wherein the second machine learning model output comprises a first authorization decision;
- classify the inference sample into a first training sample group of a plurality of training sample groups based on the second machine learning model output;
- determine a real-time feature similarity point distribution associated with the first training sample group based on each training sample of the first training sample group, wherein the real-time feature similarity point distribution includes a feature similarity point for each training sample of the first training sample group and the inference sample, and wherein, the program instructions that cause the at least one processor to determine the real-time feature similarity point distribution, cause the at least one processor to:
- determine a feature similarity point for the inference sample and each training sample of the first training sample group;
- automatically provide an indication of whether the second machine learning model is aligned with the first machine learning model based on a comparison of a historical feature similarity point distribution to the real-time feature similarity point distribution, wherein the historical feature similarity point distribution of each training sample group includes the feature similarity point for each pair of training samples in the first training sample group;
- provide a second plurality of historical payment transactions in response to an indication that the second machine learning model is not aligned with the first machine learning model, wherein transaction data associated with each historical payment transaction of the second plurality of historical payment transactions comprises a second plurality of transaction parameters and an authorization decision, and wherein the second plurality of historical payment transactions comprises payment transactions having an authorization decision different from the first authorization decision;
- determine at least one impact parameter of the first plurality of transaction parameters;
- and
- generate a modified first machine learning model using a loss function that depends on the at least one impact parameter, wherein, the program instructions that cause the at least one processor to generate the modified first machine learning model, cause the at least one processor to:
- modify one or more weights or parameters of the first machine learning model using the loss function that depends on the at least one impact parameter until the second plurality of historical payment transactions no longer cause the first machine learning model to generate authorization decisions different from the first authorization decision.
10 more elements
Dependencies are inferred from the claim text.