# US 2018/0322406 A1 · Systems and methods for providing machine learning model explainability information > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2018/0322406 A1 - Document ID: US20180322406A1 - Status: Published application - Kind: A1 - Application number: 15/970,626 - Filed: 2018-05-03 - Published: 2018-11-08 - Earliest claimed: 2017-05-04 - Assignees: ZestFinance, Inc. - Applicants: ZestFinance, Inc. - Inventors: Douglas C. Merrill; Michael Edward Ruberry; Ozan Sayin; Bojan Tunguz; Lin Song; Esfandiar Alizadeh; Melanie Eunique DeBruin; Yachen Yan; Derek Wilcox; John Candido; Benjamin Anthony Solecki; Jiahuan He; Jerome Louis Budzik - Web page: https://patentgrep.com/patent/US20180322406A1 ## CPC - G06N 5/045: Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence - G06N 20/00: Machine learning - G06N 20/20: Ensemble learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Systems and methods for generating explanation information for a result of an application system. Explanation configuration is generated based on received user input. Responsive to an explanation generation event, a plurality of modified input variable value sets are generated for a first applicant by using the explanation configuration. For each modified input variable value set: a request is provided to a first application system for generation of a result for the modified input variable value set, and a result is received for the modified input variable value set. At least one input variable value is selected based on a comparison between a first result of a first input variable value set of the first applicant and results for the modified input variable value set. Explanation information is generated for the first result by using human-readable description information for each selected input variable value, in accordance with the explanation configuration. ## Claims ### Claim 1 A method comprising: an explanation system: generating explanation configuration information based on received user input; responsive to an explanation generation event, generating a plurality of modified input variable value sets for a first applicant by using the explanation configuration information; for each modified input variable value set: providing a request to a first application system of the explanation generation event for generation of a result for the modified input variable value set, and receiving a result for the modified input variable value set from the application system as a response to the request; selecting at least one input variable value based on a comparison between a first result of a first input variable value set of the first applicant and results for the modified input variable value sets; and generating explanation information for the first result by using human-readable description information for each selected input variable value. ### Claim 2 A method comprising: an explanation system: generating explanation configuration information based on received user input; responsive to an explanation generation event, generating a plurality of modified input variable value sets for a first applicant by using the explanation configuration information; for each modified input variable value set: providing a request to a first application system of the explanation generation event for generation of a result for the modified input variable value set, and receiving a result for the modified input variable value set from the application system as a response to the request; selecting at least one input variable based on a comparison between a first result of a first input variable value set of the first applicant and results for the modified input variable value sets; and generating explanation information for the first result by using human-readable description information for each selected input variable. ### Claim 3 (depends on 1) The method of claim 1, wherein the user input is received from an operator device via one of an API and a user interface module of the explanation system, and wherein the user input includes at least one of: user-selection of variables, user-selection of variable values for at least one specified variable, user-selection of variable combinations, user-specified transformation information, user-specified impactful variable selection criteria, user-specified impactful variable value selection criteria for at least one specified variable, user-specified human-readable description information for at least one variable, and user-specified human-readable description information for at least one variable value for a specified variable. ### Claim 4 (depends on 1) The method of claim 1, wherein the explanation generation event is reception of a first explanation request from the first application system. ### Claim 5 (depends on 1) The method of claim 1, wherein the explanation generation event is identification of a denial decision generated by the first application system. ### Claim 6 (depends on 1) The method of claim 1, wherein each input variable value is selected by using the explanation configuration information. ### Claim 7 (depends on 1) The method of claim 1, wherein the explanation information is generated in accordance with the explanation configuration information. ### Claim 8 (depends on 1) The method of claim 1, further comprising: the explanation system providing the generated explanation information to the first application system by using a callback of the explanation generation event. ### Claim 9 (depends on 1) The method of claim 1, further comprising: the explanation system providing the generated explanation information to a device of the first applicant based on information of the explanation generation event. ### Claim 10 (depends on 1) The method of claim 1, wherein the explanation configuration information is generated prior to receiving the explanation generation event. ### Claim 11 (depends on 1) A method of claim 1, wherein the explanation system receives the explanation generation event from the first application system via an API of the explanation system. ### Claim 12 (depends on 1) The method of claim 1, further comprising: the first application system using a first model to generate the first result of the first input variable value set by using financial data of the first applicant and non-financial data of the first applicant. ### Claim 13 (depends on 12) The method of claim 12, wherein the non-financial data of the first applicant includes at least one of: social media data, search history data, browsing data, telephone and device data, application utilization data, educational history data, GPS data, device information, and sensor data. ### Claim 14 (depends on 1) The method of claim 1, further comprising: the first application system generating a first applicant denial decision for the first applicant by using the first result for the first input variable value set of the first applicant, wherein the explanation system receives the explanation generation event responsive to generation of first applicant denial decision, and wherein the explanation information for the first result includes an explanation for denial of an application of the first applicant. ### Claim 15 (depends on 1) The method of claim 1, further comprising: the explanation system providing the generated explanation information to the first application system by using a callback of the first application system that is associated with the explanation generation event; the first application system receiving the generated explanation information; and the first application system providing the generated explanation information to an applicant device of the first applicant. ### Claim 16 (depends on 15) The method of claim 15, further comprising: the first application system receiving a first application request from the applicant device, wherein the first application system provides the generated explanation information to the applicant device as a response to the first application request. ### Claim 17 (depends on 15) The method of claim 15, further comprising: the first application system receiving a first application request from the applicant device via a user interface module of the first application system, wherein the first application system provides the generated explanation information to the applicant device as a response to the first application request via the user interface module. ### Claim 18 (depends on 17) The method of claim 17, wherein the user interface module of the first application system includes an application server, and wherein the first application system is communicatively coupled to the first applicant device via the Internet, and wherein the first application request is a request for approval of a loan application of the first applicant. ### Claim 19 (depends on 2) The method of claim 2, wherein each result is a decision value generated by a decision generation system of the first application system. ### Claim 20 (depends on 2) The method of claim 2, wherein each result is a score generated by a score generation system of the first application system. In some embodiments, the first result is a decision value generated by a decision generation system of the first application system.