# US 12,223,453 B2 · Method for scoring confidence of an algorithmically proposed risk > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,453 B2 - Document ID: US12223453B2 - Status: Granted - Kind: B2 - Application number: 17/727,109 - Filed: 2022-04-22 - Published: 2023-10-26 - Granted: 2025-02-11 - Earliest claimed: 2022-04-22 - Assignees: Capital One Services, LLC - Applicants: Capital One Services, LLC - Inventors: Samuel Rapowitz; Joshua Peters; Haytham Yaghi; Evelyn Burke; Caroline Williams; Shabnam Kousha; Maxwell Gotsch - Web page: https://patentgrep.com/patent/US12223453B2 ## CPC - G06Q 10/0635: Risk analysis of enterprise or organisation activities - G06N 20/00: Machine learning - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Disclosed herein are system, method, and computer program product embodiments for training and deploying a machine learning model to generate an assessment of risks and mitigations in response to a novel initiative request. After generating labeled data from a corpus of prior risk assessments, a machine learning model may be trained to programmatically generate a risk assessment in response to a novel initiative request. A risk auditor may subsequently review the risk assessment generated using the machine learning model to provide various feedback reflecting the accuracy of the risks and mitigations. The machine learning model may then be retrained based on the feedback provided by the risk auditor to provide more accurate risks and mitigations in response to future initiatives. ## Claims ### Claim 1 A computer implemented method, comprising: training a machine learning model with input materials and output materials from a plurality of prior risk assessments, wherein the input materials comprise text-based documents from prior manually processed initiative requests, the text-based documents comprising a plurality of intake forms, a plurality of answered questions, and a plurality of slide presentations, and wherein the output materials comprise risks and mitigations previously generated in response to the input materials; generating, by one or more processors, using the machine learning model, a risk assessment for an initiative request received from an intent requestor, wherein the risk assessment comprises one or more risks and one or more mitigations; displaying a risk assessment interface that presents summary information for the risk assessment, the one or more risks in association with the one or more mitigations, and a feedback interface that allows the intent requestor to provide feedback for each risk in the one or more risks; receiving the feedback from the intent requestor, wherein the feedback comprises a relevancy indicator and an accuracy indicator; and retraining the machine learning model based on the relevancy indicator and the accuracy indicator to cause the machine learning model to provide more accurate results in response to a future initiative request. ### Claim 2 (depends on 1) The method of claim 1, further comprising: identifying a risk auditor; displaying the risk assessment interface to the risk auditor; receiving additional feedback from the risk auditor; and updating the machine learning model based on the additional feedback. ### Claim 3 (depends on 1) The method of claim 1, wherein the risk assessment further comprises a severity indicator associated with a risk in the one or more risks. ### Claim 4 (depends on 1) The method of claim 1, further comprising: applying sentiment analysis to the feedback to extract a polarity of the feedback; and generating the relevancy indicator based on the polarity. ### Claim 5 (depends on 1) The method of claim 1, further comprising: determining, a risk advisor associated with a mitigation in the one or more mitigations; generating a summary of the initiative request; and displaying a summary page in the risk assessment interface to the risk advisor comprising the summary, the one or more risks, and the one or more mitigations. ### Claim 6 (depends on 1) The method of claim 1, wherein the relevancy indicator comprises a selection of relevant or irrelevant. ### Claim 7 (depends on 1) The method of claim 1, wherein the accuracy indicator quantifies the accuracy of the risk and comprises a numerical score or a letter grade. ### Claim 8 A system, comprising: a memory; and at least one processor coupled to the memory and configured to: train a machine learning model with input materials and output materials from a plurality of prior risk assessments, wherein the input materials comprise text-based documents from prior manually processed initiative requests, the text-based documents comprising a plurality of intake forms, a plurality of answered questions, and a plurality of slide presentations, and wherein the output materials comprise risks and mitigations previously generated in response to the input materials; generate using the machine learning model, a risk assessment for an initiative request received from an intent requestor, wherein the risk assessment comprises one or more risks and one or more mitigations; display a risk assessment interface that presents summary information for the risk assessment, the one or more risks in association with the one or more mitigations, and a feedback interface that allows the intent requestor to provide feedback for each risk in the one or more risks; receive the feedback from the intent requestor, wherein the feedback comprises a relevance indicator and an accuracy indicator; and retrain the machine learning model based on the relevancy indicator and the accuracy indicator to cause the machine learning model to provide more accurate results in response to future initiative request. ### Claim 9 (depends on 8) The system of claim 8, the at least one processor further configured to: identify a risk auditor; display the risk assessment interface to the risk auditor; receive additional feedback from the risk auditor; and update the machine learning model based on the additional feedback. ### Claim 10 (depends on 8) The system of claim 8, wherein the risk assessment further comprises a severity indicator associated with a risk in the one or more risks. ### Claim 11 (depends on 8) The system of claim 8, the at least one processor further configured to: apply sentiment analysis to the feedback to extract a polarity of the feedback; and generate the relevancy indicator based on the polarity. ### Claim 12 (depends on 8) The system of claim 8, the at least one processor further configured to: determine a risk advisor associated with a mitigation in the one or more mitigations; generate a summary of the initiative request; and display a summary page in the risk assessment interface to the risk advisor comprising the summary, the one or more risks, and the one or more mitigations. ### Claim 13 (depends on 8) The system of claim 8, wherein the relevancy indicator comprises a selection of relevant or irrelevant. ### Claim 14 (depends on 8) The system of claim 8, wherein the accuracy indicator quantifies the accuracy of the risk and comprises a numerical score or a letter grade. ### Claim 15 A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising: training a machine learning model with input materials and output materials from a plurality of prior risk assessments, wherein the input materials comprise text-based documents from prior manually processed initiative requests, the text-based documents comprising a plurality of intake forms, a plurality of answered questions, and a plurality of slide presentations, wherein the output materials comprise risks and mitigations previously generated in response to the input materials; generating, using the machine learning model, a risk assessment for an initiative request received from an intent requestor, wherein the input materials comprise text-based documents from prior manually processed initiative requests, and wherein the risk assessment comprises one or more risks and one or more mitigations; displaying a risk assessment interface that presents summary information for the risk assessment, the one or more risks in association with the one or more mitigations, and a feedback interface that allows the intent requestor to provide feedback for each risk in the one or more risks; receiving the feedback from the intent requestor, wherein the feedback comprises a relevancy indicator and an accuracy indicator; and retraining the machine learning model based on the relevancy indicator and the accuracy indicator to cause the machine learning model to provide more accurate results in response to a future initiative request. ### Claim 16 (depends on 15) The non-transitory computer-readable device of claim 15, the operations further comprising: identifying a risk auditor; displaying the risk assessment interface to the risk auditor; receiving additional feedback from the risk auditor; and updating the machine learning model based on the additional feedback. ### Claim 17 (depends on 15) The non-transitory computer-readable device of claim 15, wherein the risk assessment further comprises a severity indicator associated with a risk in the one or more risks. ### Claim 18 (depends on 15) The non-transitory computer-readable device of claim 15, the operations further comprising: applying sentiment analysis to the feedback to extract a polarity of the feedback; and generating the relevancy indicator based on the polarity. ### Claim 19 (depends on 15) The non-transitory computer-readable device of claim 15, the operations further comprising: determining, a risk advisor associated with a mitigation in the one or more mitigations; generating a summary of the initiative request; and displaying a summary page in the risk assessment interface comprising the summary, the one or more risks, and the one or more mitigations. ### Claim 20 (depends on 15) The non-transitory computer-readable device of claim 15, wherein the relevancy indicator comprises a selection of relevant or irrelevant.