# US 12,002,258 B2 · System and method for mitigating bias in classification scores generated by machine learning models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,002,258 B2 - Document ID: US12002258B2 - Status: Granted - Kind: B2 - Application number: 16/891,989 - Filed: 2020-06-03 - Published: 2021-12-09 - Granted: 2024-06-04 - Earliest claimed: 2020-06-03 - Assignees: Discover Financial Services - Applicants: Discover Financial Services - Inventors: Alexey Miroshnikov; Konstandinos Kotsiopoulos; Arjun Ravi Kannan; Raghu Kulkarni; Steven Dickerson - Web page: https://patentgrep.com/patent/US12002258B2 ## CPC - G06V 10/82: using neural networks - G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} - G06F 18/23: Clustering techniques - G06N 20/00: Machine learning - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/776: Validation; Performance evaluation - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A method, system, and computer-readable medium are disclosed for detecting and mitigating bias in a trained machine learning model. The method includes the steps of: training the model based on a training data set; detecting bias in the model relative to a protected class; identifying one or more groups of input variables that contribute to the bias; and mitigating bias in the model. Mitigating the bias is performed by constructing a post-processed score function that either (a) neutralizes or partially neutralizes one or more groups of input variables in the input vector of the model, or (b) utilizes a fair score approximation of the model to project the distributions for the protected class and/or the unprotected class to substantially match. In an embodiment, detecting bias in the trained model is performed by comparing distribution for two or more subpopulations based on a distance metric, such as a Wasserstein distance. ## Claims ### Claim 1 A method carried out by a computing system comprising one or more processors and at least one non-transitory computer-readable medium, the method comprising: training a machine learning model by carrying out a machine learning process on a training data set, wherein the trained machine learning model is configured to (i) receive an input vector comprising respective values for a given set of input variables and (ii) based on an evaluation of the received input vector, output a score for use in rendering a given type of classification decision; detecting bias in the trained machine learning model relative to a protected class by (i) inputting a first set of input vectors associated with an unprotected class into the trained machine learning model and thereby producing a first set of scores associated with the unprotected class (ii) inputting a second set of input vectors associated with the protected class into the trained machine learning model and thereby producing a second set of scores associated with the protected class, (iii) performing a comparison between the first set of scores and the second set of scores, and (iv) based on the comparison, determining that the trained machine learning model exhibits a threshold level of bias; in response to determining that the trained machine learning model exhibits a threshold level of bias, identifying one or more input variable groups that contribute to the bias by (i) dividing the trained machine learning model's given set of input variables into a plurality of input variable groups, wherein each respective input variable group includes a respective subset of the given set of input variables, (ii) determining respective bias contribution values f or the plurality of input variable groups using a score explainer function, and (iii) based on the respective bias contribution values that are determined for the plurality of input variable groups, identifying the one or more input variable groups that contribute to the bias; mitigating bias in the trained machine learning model by producing a post-processed version of the trained machine learning model that comprises a respective transformation in place of each of the identified one or more input variable groups, wherein the post-processed version of the trained machine learning model is configured to (i) receive an input vector comprising respective values for a given set of input variables, (ii) for each of the identified one or more input variable groups, apply the respective transformation to the respective value for the input variable group, and (iii) based on an evaluation of the transformed input vector, output a score for use in rendering the given type of classification decision; and using the post-processed version of the trained machine learning model to render the given type of classification decision for a given input vector by (i) executing the post-processed version of the trained machine learning model, (ii) while executing the post-processed version of the trained machine learning model, inputting the given input vector into the post-processed version of the trained machine learning model and thereby causing the post-processed version of the trained machine learning model to output a given score for the given input vector, and (iii) comparing the given score to a classification threshold for the given type of classification decision. ### Claim 2 (depends on 1) The method of claim 1, wherein performing the comparison between the first set of scores and the second set of scores comprises: comparing distributions of the first and second sets of scores based on a Wasserstein distance metric. ### Claim 3 (depends on 1) The method of claim 1, wherein the bias is separated into a positive bias and a negative bias. ### Claim 4 (depends on 1) The method of claim 1, wherein dividing the trained machine learning model's given set of input variables into the plurality of input variable groups comprises: dividing the trained machine learning model's given set of input variables into the plurality of input variable groups based on a clustering algorithm. ### Claim 5 (depends on 1) The method of claim 1, wherein the respective bias contribution values for the plurality of input variable groups are based on group partial dependence plot (PDP) metrics. ### Claim 6 (depends on 1) The method of claim 1, wherein the respective bias contribution values for the plurality of input variable groups are based on Shapley Additive Explanation (SNAP) metrics. ### Claim 7 (depends on 1) The method of claim 1, wherein the respective transformation for at least one respective input variable group of the identified one or more input variable groups functions to at least partially neutralize the respective input variable group. ### Claim 8 (depends on 1) The method of claim 1, wherein the respective transformation for at least one respective input variable group of the identified one or more input variable groups functions to compress each input variable in the respective input variable group. ### Claim 9 (depends on 8) The method of claim 8, wherein the respective transformation for the at least one respective input variable group functions to compress the distribution of each input variable in the respective input variable group towards a median value for the input variable. ### Claim 10 A computing system comprising: at least one non-transitory computer-readable medium; one or more processors; and program instructions stored on the at least one non-transitory computer-readable medium that are executable by the one or more processors such that the computing system is configured to: train a machine learning model by carrying out a machine learning process on a training data set, wherein the trained machine learning model is configured to (i) receive an input vector comprising respective values for a given set of input variables and (ii) based on an evaluation of the received input vector, output a score for use in rendering a given type of classification decision; detect bias in the trained machine learning model relative to a protected class by (i) inputting a first set of input vectors associated with an unprotected class into the trained machine learning model and thereby producing a first set of scores associated with the unprotected class (ii) inputting a second set of input vectors associated with the protected class into the trained machine learning model and thereby producing a second set of scores associated with the protected class, (iii) performing a comparison between the first set of scores and the second set of scores, and (iv) based on the comparison, determining that the trained machine learning model exhibits a threshold level of bias; in response to determining that the trained machine learning model exhibits a threshold level of bias, identify one or more input variable groups that contribute to the bias by (i) dividing the trained machine learning model's given set of input variables into a plurality of input variable groups, wherein each respective input variable group includes a respective subset of the given set of input variables, (ii) determining respective bias contribution values for the plurality of input variable groups using a score explainer function, and (iii) based on the respective bias contribution values that are determined for the plurality of input variable groups, identifying the one or more input variable groups that contribute to the bias; mitigate bias in the trained machine learning model by producing a post-processed version of the trained machine learning model that comprises a respective transformation in place of each of the identified one or more input variable groups, wherein the post-processed version of the trained machine learning model is configured to (i) receive an input vector comprising respective values for a given set of input variables, (ii) for each of the identified one or more input variable groups, apply the respective transformation to the respective value for the input variable group, and (iii) based on an evaluation of the transformed input vector, output a score for use in rendering the given type of classification decision; and use the post-processed version of the trained machine learning model to render the given type of classification decision for a given input vector by (i) executing the post-processed version of the trained machine learning model, (ii) while executing the post-processed version of the trained machine learning model, inputting the given input vector into the post-processed version of the trained machine learning model and thereby causing the post-processed version of the trained machine learning model to output a given score for the given input vector, and (iii) comparing the given score to a classification threshold for the given type of classification decision. ### Claim 11 (depends on 10) The computing system of claim 10, wherein performing the comparison between the first set of scores and the second set of scores comprises: comparing distributions of the first and second sets of scores based on a Wasserstein distance metric. ### Claim 12 (depends on 10) The computing system of claim 10, wherein dividing the trained machine learning model's given set of input variables into the plurality of input variable groups comprises: dividing the trained machine learning model's given set of input variables into the plurality of input variable groups based on a clustering algorithm. ### Claim 13 (depends on 10) The computing system of claim 10, wherein the respective bias contribution values for the plurality of input variable groups are based on group partial dependence plot (PDP) metrics. ### Claim 14 (depends on 10) The computing system of claim 10, wherein the respective transformation for at least one respective input variable group of the identified one or more input variable groups at least partially neutralizes the respective input variable group. ### Claim 15 A non-transitory computer-readable media storing computer instructions that, when executed by one or more processors, cause a computer system to perform functions comprising: training a machine learning model by carrying out a machine learning process on a training data set, wherein the trained machine learning model is configured to (i) receive an input vector comprising respective values for a given set of input variables and (ii) based on an evaluation of the received input vector, output a score for use in rendering a given type of classification decision; detecting bias in the trained machine learning model relative to a protected class by (i) inputting a first set of input vectors associated with an unprotected class into the trained machine learning model and thereby producing a first set of scores associated with the unprotected class (ii) inputting a second set of input vectors associated with the protected class into the trained machine learning model and thereby producing a second set of scores associated with the protected class, (iii) performing a comparison between the first set of scores and the second set of scores, and (iv) based on the comparison, determining that the trained machine learning model exhibits a threshold level of bias; in response to determining that the trained machine learning model exhibits a threshold level of bias, identifying one or more input variable groups that contribute to the bias by (i) dividing the trained machine learning model's given set of input variables into a plurality of input variable groups, wherein each respective input variable group includes a respective subset of the given set of input variables, (ii) determining respective bias contribution values for the plurality of input variable groups using a score explainer function, and (iii) based on the respective bias contribution values that are determined for the plurality of input variable groups, identifying the one or more input variable groups that contribute to the bias; mitigating bias in the trained machine learning model by producing a post-processed version of the trained machine learning model that comprises a respective transformation in place of each of the identified one or more input variable groups, wherein the post-processed version of the trained machine learning model is configured to (i) receive an input vector comprising respective values for a given set of input variables, (ii) for each of the identified one or more input variable groups, apply the respective transformation to the respective value for the input variable group, and (iii) based on an evaluation of the transformed input vector, output a score for use in rendering the given type of classification decision; and using the post-processed version of the trained machine learning model to render the given type of classification decision for a given input vector by (i) executing the post-processed version of the trained machine learning model, (ii) while executing the post-processed version of the trained machine learning model, inputting the given input vector into the post-processed version of the trained machine learning model and thereby causing the post-processed version of the trained machine learning model to output a given score for the given input vector, and (iii) comparing the given score to a classification threshold for the given type of classification decision. ### Claim 16 (depends on 1) The method of claim 1, wherein the given type of classification decision comprises a type of classification decision that is subject to review under the Equal Credit Opportunity Act (ECOA). ### Claim 17 (depends on 10) The computing system of claim 10, wherein the respective bias contribution values for the plurality of input variable groups are based on Shapley Additive Explanation (SHAP) metrics. ### Claim 18 (depends on 10) The computing system of claim 10, wherein the respective transformation for at least one respective input variable group of the identified one or more input variable groups functions to compress each input variable in the respective input variable group. ### Claim 19 (depends on 10) The computing system of claim 10, wherein the given type of classification decision comprises a type of classification decision that is subject to review under the Equal Credit Opportunity Act (ECOA). ### Claim 20 (depends on 10) The computing system of claim 10, wherein the respective bias contribution values for the plurality of input variable groups comprise values that separately quantify positive bias and negative bias.