# US 11,907,882 B1 · Model validation of credit risk > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,882 B1 - Document ID: US11907882B1 - Status: Granted - Kind: B1 - Application number: 17/895,140 - Filed: 2022-08-25 - Granted: 2024-02-20 - Earliest claimed: 2018-05-01 - Assignees: Wells Fargo Bank, N.A. - Applicants: Wells Fargo Bank, N.A. - Inventors: Vijayan Narayana Nair; Agus Sudjianto; Weicheng Liu; Jie Chen; Kevin David Oden - Web page: https://patentgrep.com/patent/US11907882B1 ## CPC - G06Q 10/0635: Risk analysis of enterprise or organisation activities - G06N 20/00: Machine learning - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract The innovation disclosed and claimed herein, in one aspect thereof, comprises systems and methods of validating models guided by machine learning algorithms. The innovation can begin by receiving a risk model for validation having multiple sets of data. A first data set is selected from as an input. Outputs are generated for validation. One output can be generating a second set of analysis results using a comparable algorithm to the risk model. Another output can be generating a second set of variables and transformations using a machine learning algorithm and an untransformed set of the selected variables to assess the set of selected transformations. Another output can be generating a third set of variables using one or more machine learning algorithms and an extended feature set of variables to assess the selected variables. The outputs are compared to the analysis results, coefficients, selected variables, and selected transformations. A report of the comparison is generated. ## Claims ### Claim 1 A computer-implemented method, comprising: identifying a risk model for validation, the risk model including a first data set and a plurality of result data sets corresponding to the first data set and the risk model, the plurality of result data sets including a first set of analysis results, a first set of coefficients, a first set of transformations, and a first set of variables including a first set of transformed variables and a first set of untransformed variables; performing a replication validation of the risk model, the replication validation of the risk model including: analyzing the first data set with a logistic regression model to produce a second set of analysis results including a second set of coefficients, and comparing at least one portion of the first set of analysis results to at least one portion of the second set of analysis results; performing a transformation benchmarking validation of the risk model, the transformation benchmarking validation of the risk model including: inputting the first set of untransformed variables and the first data set into a first machine learning algorithm to produce a second set of transformations and a second set of transformed variables, and comparing at least one portion of the first set of transformed variables to at least one portion of the second set of transformed variables; performing a predictive performance benchmarking of the risk model, the predictive performance benchmarking of the risk model including: identifying an extended feature set of variables; selecting a second set of variables from the extended feature set of variables with a second machine learning algorithm, and comparing at least one portion of the first set of variables to at least one portion of the second set of variables; and generating a digital report based on comparison of the at least one portion of the first set of analysis results to the at least one portion of the second set of analysis results, comparison of the at least one portion of the first set of transformed variables to the at least one portion of the second set of transformed variables, and comparison of the at least one portion of the first set of variables to the at least one portion of the second set of variables. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the first set of variables was selected from a first feature set of variables having fewer variables than the extended feature set of variables. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein comparing the at least one portion of the first set of analysis results to the at least one portion of the second set of analysis results includes: calculating deviation percentages between the first set of coefficients and the second set of coefficients; and flagging a subset of coefficients in the first set of coefficients that deviate more than a predetermined threshold deviation from the second set of coefficients. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein comparing the at least one portion of the first set of transformed variables to the at least one portion of the second set of transformed variables include: selecting a subset of the second set of transformed variables and a subset of the first set of transformed variables, wherein each subset is a selection of most dominant variables of each set; and computing diagnostic functions between the subsets. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the logistic regression model is comparable to the risk model. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the risk model comprises a predictive model used for determining risk associated with providing services or credit to a customer. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the digital report includes at least one receiver operating characteristic (ROC) curve or an area under an ROC. ### Claim 8 A non-transitory computer readable medium for validating a model, the non-transitory computer readable medium having instructions to control one or more processors configured to: identify a risk model for validation, the risk model including a first data set and a plurality of result data sets corresponding to the first data set and the risk model, the plurality of result data sets including a first set of analysis results, a first set of coefficients, a first set of transformations, and a first set of variables including a first set of transformed variables and a first set of untransformed variables; perform a replication validation of the risk model, the replication validation of the risk model including: analyzing the first data set with a logistic regression model to produce a second set of analysis results including a second set of coefficients, and comparing at least one portion of the first set of analysis results to at least one portion of the second set of analysis results; perform a transformation benchmarking validation of the risk model, the transformation benchmarking validation of the risk model including: inputting the first set of untransformed variables and the first data set into a first machine learning algorithm to produce a second set of transformations and a second set of transformed variables, and comparing at least one portion of the first set of transformed variables to at least one portion of the second set of transformed variables; perform a predictive performance benchmarking of the risk model, the predictive performance benchmarking of the risk model including: identifying an extended feature set of variables; selecting a second set of variables from the extended feature set of variables with a second machine learning algorithm, and comparing at least one portion of the first set of variables to at least one portion of the second set of variables; and generate a digital report based on comparison of the at least one portion of the first set of analysis results to the at least one portion of the second set of analysis results, comparison of the at least one portion of the first set of transformed variables to the at least one portion of the second set of transformed variables, and comparison of the at least one portion of the first set of variables to the at least one portion of the second set of variables. ### Claim 9 (depends on 8) The non-transitory computer readable medium of claim 8, wherein the first set of variables was selected from a first feature set of variables having fewer variables than the extended feature set of variables. ### Claim 10 (depends on 8) The non-transitory computer readable medium of claim 8, wherein comparing the at least one portion of the first set of analysis results to the at least one portion of the second set of analysis results includes: calculating deviation percentages between the first set of coefficients and the second set of coefficients; and flagging a subset of coefficients in the first set of coefficients that deviate more than a predetermined threshold deviation from the second set of coefficients. ### Claim 11 (depends on 8) The non-transitory computer readable medium of claim 8, wherein comparing the at least one portion of the first set of transformed variables to the at least one portion of the second set of transformed variables include: selecting a subset of the second set of transformed variables and a subset of the first set of transformed variables, wherein each subset is a selection of most dominant variables of each set; and computing diagnostic functions between the subsets. ### Claim 12 (depends on 8) The non-transitory computer readable medium of claim 8, wherein the logistic regression model is comparable to the risk model. ### Claim 13 (depends on 8) The non-transitory computer readable medium of claim 8, wherein the risk model comprises a predictive model used for determining risk associated with providing services or credit to a customer. ### Claim 14 (depends on 8) The non-transitory computer readable medium of claim 8, wherein the digital report includes at least one receiver operating characteristic (ROC) curve or an area under an ROC. ### Claim 15 A system for validating a model, comprising: one or more hardware processors; a development component using at least one of the one or more hardware processors to identify a risk model for validation, the risk model including a first data set and a plurality of result data sets corresponding to the first data set and the risk model, the plurality of result data sets including a first set of analysis results, a first set of coefficients, a first set of transformations, and a first set of variables including a first set of transformed variables and a first set of untransformed variables; a replication component using at least one of the one or more hardware processors to perform a replication validation of the risk model, the replication validation of the risk model to: analyze the first data set with a logistic regression model to produce a second set of analysis results including a second set of coefficients, and compare at least one portion of the first set of analysis results to at least one portion of the second set of analysis results; a transformation benchmark component using at least one of the one or more hardware processors to perform a transformation benchmarking validation of the risk model, the transformation benchmarking validation of the risk model to: input the first set of untransformed variables and the first data set into a first machine learning algorithm to produce a second set of transformations and a second set of transformed variables, and compare at least one portion of the first set of transformed variables to at least one portion of the second set of transformed variables; a predictive performance benchmark component using at least one of the one or more hardware processors to perform a predictive performance benchmarking of the risk model, the predictive performance benchmarking of the risk model to: identify an extended feature set of variables; select a second set of variables from the extended feature set of variables with a second machine learning algorithm, and compare at least one portion of the first set of variables to at least one portion of the second set of variables; and a report component using at least one of the one or more hardware processors to generate a digital report based on comparison of the at least one portion of the first set of analysis results to the at least one portion of the second set of analysis results, comparison of the at least one portion of the first set of transformed variables to the at least one portion of the second set of transformed variables, and comparison of the at least one portion of the first set of variables to the at least one portion of the second set of variables. ### Claim 16 (depends on 15) The system of claim 15, wherein the first set of variables was selected from a first feature set of variables having fewer variables than the extended feature set of variables. ### Claim 17 (depends on 15) The system of claim 15, wherein comparing the at least one portion of the first set of analysis results to the at least one portion of the second set of analysis results includes: calculating deviation percentages between the first set of coefficients and the second set of coefficients; and flagging a subset of coefficients in the first set of coefficients that deviate more than a predetermined threshold deviation from the second set of coefficients. ### Claim 18 (depends on 15) The system of claim 15, wherein comparing the at least one portion of the first set of transformed variables to the at least one portion of the second set of transformed variables include: selecting a subset of the second set of transformed variables and a subset of the first set of transformed variables, wherein each subset is a selection of most dominant variables of each set; and computing diagnostic functions between the subsets. ### Claim 19 (depends on 15) The system of claim 15, wherein the risk model comprises a predictive model used for determining risk associated with providing services or credit to a customer. ### Claim 20 (depends on 15) The system of claim 15, wherein the digital report includes at least one receiver operating characteristic (ROC) curve or an area under an ROC.