Back to results
US 12,585,969 B2Granted
Generating confidence scores for machine learning model predictions
AssigneeOracle International Corporation
InventorsMatthew Charles Rowe, Alberto Polleri, Rhys David Green
Life of this patent
Estimating the term…
Abstract
Techniques for generating confidence scores for machine learning predictions are disclosed. The confidence score for a predicted label corresponding to a target data point is based at least in part on how well the machine learning model predicts labels for other data points that are similar to the target data point. The system uses k data points, closest to the target data point, with known labels to compute the confidence score of a predicted label for the target data point. The accuracy of the predictions and the distance of each of the k data points from the target data point are used to compute a confidence score for a label predicted for the target data point.
Claims23
A non-transitory computer readable medium comprising instructions which, when executed by one or more hardware processors, causes performance of operations comprising:
- obtaining, by a data analysis platform executing on a computer comprising the one or more hardware processors, a dataset including (a) a plurality of data points in a multidimensional space comprising three or more dimensions, the plurality of data points corresponding to a respective plurality of system operations, and (b) a plurality of labels corresponding respective to the plurality of data points, the plurality of labels corresponding to success or failure of the respective plurality of system operations;
- training, by a machine learning engine of the data analysis platform using a first subset of the dataset, a machine learning model to predict success or failure of target system operations represented by respective target data points in the multidimensional space;
- executing, by the data analysis platform, a performance-based retraining of the machine learning model at least by:
- accessing a first set of system operation data corresponding to a first system operation;
- generating, based on the first set of system operation data, a first target data point in the multidimensional space, the first target data point comprising values corresponding to the three or more dimensions in the multidimensional space;
- inputting the first target data point into the machine learning model to generate a first predicted-value representing success or failure of the first system operation corresponding to the first target data point;
- determining a first confidence score for the first predicted value at least by:
- identifying a first subset of the plurality of data points in the multidimensional space corresponding to a second subset of the dataset;
- identifying a first set of k data points, from the first subset of the plurality of data points, closest to the first target data point;
- responsive to identifying the first set of k data points as being closest to the first target data point:
- inputting each of the first set of k data points into the machine learning model to predict a first set of k values, the first set of k data points corresponding respectively to a first set of known labels indicating success or failure;
- determining a first neighbor accuracy value for the machine learning model based on:
- (a) determining whether each of the first set of k predicted values was a correct prediction or an incorrect prediction by comparing the first set of k predicted values to the first set of known labels for the first set of k data points, wherein determining whether each of the first set of k predicted values was a correct prediction or an incorrect prediction corresponds to a first level of aleatoric uncertainty associated with the dataset for training the machine learning model;
- and
- (b) determining a first distance of each of the first set of k data points to the first target data point in the multidimensional space, wherein determining the first distance of each of the first set of k data points to the first target data point corresponds to a second level of epistemic uncertainty associated with the dataset for training the machine learning model;
- and
- computing the first confidence score for the first predicted value based at least on the first neighbor accuracy value;
- calculating, by the data analysis platform, a plurality of confidence scores associated with a plurality of target data points;
- calculating, by the data analysis platform, a percentage of the plurality of confidence scores that meet a threshold confidence score value;
- and
- determining that the percentage of the plurality of confidence scores that meet the threshold confidence score value does not meet a threshold percentage;
- and
- executing, by the data analysis platform, a retraining process for retraining the machine learning model.
20 more elements
A method comprising:
- obtaining, by a data analysis platform executing on a computer comprising the one or more hardware processors, a dataset including (a) a plurality of data points in a multidimensional space comprising three or more dimensions, the plurality of data points corresponding to a respective plurality of system operations, and (b) a plurality of labels corresponding respective to the plurality of data points, the plurality of labels corresponding to success or failure of the respective plurality of system operations;
- training, by a machine learning engine of the data analysis platform using a first subset of the dataset, a machine learning model to predict success or failure of target system operations represented by respective target data points in the multidimensional space;
- executing, by the data analysis platform, a performance-based retraining of the machine learning model at least by:
- accessing a first set of system operation data corresponding to a first system operation;
- generating, based on the first set of system operation data, a first target data point in the multidimensional space, the first target data point comprising values corresponding to the three or more dimensions in the multidimensional space;
- inputting the first target data point into the machine learning model to generate a first predicted-value representing success or failure of the first system operation corresponding to the first target data point;
- determining a first confidence score for the first predicted value at least by:
- identifying a first subset of the plurality of data points in the multidimensional space corresponding to a second subset of the dataset;
- identifying a first set of k data points, from the first subset of the plurality of data points, closest to the first target data point;
- responsive to identifying the first set of k data points as being closest to the first target data point:
- inputting each of the first set of k data points into the machine learning model to predict a first set of k values, the first set of k data points corresponding respectively to a first set of known labels indicating success or failure;
- determining a first neighbor accuracy value for the machine learning model based on:
- (a) determining whether each of the first set of k predicted values was a correct prediction or an incorrect prediction by comparing the first set of k predicted values to the first set of known labels for the first set of k data points, wherein determining whether each of the first set of k predicted values was a correct prediction or an incorrect prediction corresponds to a first level of aleatoric uncertainty associated with the dataset for training the machine learning model;
- and
- (b) determining a first distance of each of the first set of k data points to the first target data point in the multidimensional space, wherein determining the first distance of each of the first set of k data points to the first target data point corresponds to a second level of epistemic uncertainty associated with the dataset for training the machine learning model;
- and
- computing the first confidence score for the first predicted value based at least on the first neighbor accuracy value;
- calculating, by the data analysis platform, a plurality of confidence scores associated with a plurality of target data points;
- calculating, by the data analysis platform, a percentage of the plurality of confidence scores that meet a threshold confidence score value;
- and
- determining that the percentage of the plurality of confidence scores that meet the threshold confidence score value does not meet a threshold percentage;
- and
- executing, by the data analysis platform, a retraining process for retraining the machine learning model.
20 more elements
A system, comprising:
- one or more processors;
- and
- memory storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
- obtaining, by a data analysis platform executing on a computer comprising the one or more hardware processors, a dataset including (a) a plurality of data points in a multidimensional space comprising three or more dimensions, the plurality of data points corresponding to a respective plurality of system operations, and (b) a plurality of labels corresponding respective to the plurality of data points, the plurality of labels corresponding to success or failure of the respective plurality of system operations;
- training, by a machine learning engine of the data analysis platform using a first subset of the dataset, a machine learning model to predict success or failure of target system operations represented by respective target data points in the multidimensional space;
- executing, by the data analysis platform, a performance-based retraining of the machine learning model at least by:
- accessing a first set of system operation data corresponding to a first system operation;
- generating, based on the first set of system operation data, a first target data point in the multidimensional space, the first target data point comprising values corresponding to the three or more dimensions in the multidimensional space;
- inputting the first target data point into the machine learning model to generate a first predicted-value representing success or failure of the first system operation corresponding to the first target data point;
- determining a first confidence score for the first predicted value at least by:
- identifying a first subset of the plurality of data points in the multidimensional space corresponding to a second subset of the dataset;
- identifying a first set of k data points, from the first subset of the plurality of data points, closest to the first target data point;
- responsive to identifying the first set of k data points as being closest to the first target data point:
- inputting each of the first set of k data points into the machine learning model to predict a first set of k values, the first set of k data points corresponding respectively to a first set of known labels indicating success or failure;
- determining a first neighbor accuracy value for the machine learning model based on:
- (a) determining whether each of the first set of k predicted values was a correct prediction or an incorrect prediction by comparing the first set of k predicted values to the first set of known labels for the first set of k data points, wherein determining whether each of the first set of k predicted values was a correct prediction or an incorrect prediction corresponds to a first level of aleatoric uncertainty associated with the dataset for training the machine learning model;
- and
- (b) determining a first distance of each of the first set of k data points to the first target data point in the multidimensional space, wherein determining the first distance of each of the first set of k data points to the first target data point corresponds to a second level of epistemic uncertainty associated with the dataset for training the machine learning model;
- and
- computing the first confidence score for the first predicted value based at least on the first neighbor accuracy value;
- calculating, by the data analysis platform, a plurality of confidence scores associated with a plurality of target data points;
- calculating, by the data analysis platform, a percentage of the plurality of confidence scores that meet a threshold confidence score value;
- and
- determining that the percentage of the plurality of confidence scores that meet the threshold confidence score value does not meet a threshold percentage;
- and
- executing, by the data analysis platform, a retraining process for retraining the machine learning model.
23 more elements
Dependencies are inferred from the claim text.