# US 12,602,277 B2 · Managing data processing system failures using hidden knowledge from predictive models for failure response generation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,277 B2 - Document ID: US12602277B2 - Status: Granted - Kind: B2 - Application number: 18/417,417 - Filed: 2024-01-19 - Published: 2025-07-24 - Granted: 2026-04-14 - Earliest claimed: 2024-01-19 - Assignees: Dell Products L.P. - Applicants: Dell Products L.P. - Inventors: Deepaganesh Paulraj; Min Gong; Ashok Narayanan Potti; Dale Wang - Web page: https://patentgrep.com/patent/US12602277B2 ## CPC - G06F 11/079: {Root cause analysis, i.e. error or fault diagnosis (in a hardware test environment G06F11/22; in a software test environment G06F11/36)} - G06F 11/0709: {in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems} - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Methods and systems for managing data processing systems are disclosed. A data processing system may include and depend on the operation of hardware and/or software components. Inference models may be implemented to predict future system infrastructure outcomes (e.g., component failures) using information recorded in logs that reflect the operation of the components. However, the models may be complex “black boxes” and may generate critical outcome predictions for downstream consumers without explanations of how the predictions are determined, resulting in downstream consumers having low confidence in the predictions. Therefore, hidden knowledge (e.g., structured knowledge attributes) of the models may be extracted and/or used to understand the underlying processes that the models use to predict the system infrastructure outcomes. The hidden knowledge may be stored in a repository and may be provided for downstream use in order to increase the likelihood of preventing and/or mitigating future data processing system failures. ## Claims ### Claim 1 A method for managing failures of data processing systems, comprising and by a data processing system manager configured to manage the data processing systems: generating a structured knowledge labeled dataset using: one or more structured knowledge attributes stored in a structured knowledge repository, the one or more structured knowledge attributes being extracted from an architecture of a trained machine learning model, the architecture controls how the trained machine learning model generates inferences, and one or more failure prediction prompts stored in a failure prediction prompt library; fine-tuning an inference model using the structured knowledge labeled dataset to obtain a trained inference model; using the trained inference model to generate a failure prediction response, the failure prediction response comprising a failure prediction and a portion of the one or more structured knowledge attributes that provide for interpretability of the failure prediction, the failure prediction being one of the inferences; and providing the failure prediction response to a user, wherein prior to generating the structured knowledge labeled dataset, the method further comprises: extracting, using one or more techniques comprising explainable artificial intelligence (AI) techniques, the one or more structured knowledge attributes from the architecture of trained machine learning model; and storing the one or more structured knowledge attributes extracted from trained machine learning model into the structured knowledge repository. ### Claim 2 (depends on 1) The method of claim 1, wherein prior to using the trained inference model to generate the failure prediction response: obtaining a failure prediction response request from the user for data stored in the structured knowledge repository, the data comprising the one or more structured knowledge attributes, and the data being usable to manage an indication of failure for a data processing system of the data processing systems. ### Claim 3 (depends on 2) The method of claim 2, wherein the one or more failure prediction prompts include at least a short readout prompt, a long readout prompt, and a knowledge-base-article-based prompt. ### Claim 4 (depends on 3) The method of claim 3, wherein the inference model is a large language model (LLM), and wherein fine-tuning the inference model using the structured knowledge labeled dataset comprises implementation of a private infrastructure fine-tuning method. ### Claim 5 (depends on 3) The method of claim 3, wherein the inference model is a large language model (LLM), and wherein fine-tuning the inference model using the structured knowledge labeled dataset comprises implementation of a public infrastructure fine-tuning method. ### Claim 6 (depends on 2) The method of claim 2, further comprising: prior to generating the failure prediction response: identifying an occurrence of the indication of failure for the data processing system; and based on the occurrence, using the inference model to obtain an indication of a root cause for the failure, the structured knowledge repository being based, at least in part, on the inference model and logs on which the inference model is based, the inference model being the trained machine learning model that generates the failure prediction and the failure prediction response, and the indication of the root cause being specified in the failure prediction. ### Claim 7 (depends on 6) The method of claim 6, further comprising: after providing the failure prediction response: assessing a likelihood of the root cause being accurate using the failure prediction response; and in an instance of the assessing where the likelihood meets a threshold: identifying at least one remediation action based on the root cause; and performing the at least one remediation action to obtain an updated data processing system to attempt to remediate the failure. ### Claim 8 (depends on 1) The method of claim 1, wherein the architecture of the trained machine learning model is hidden from a requestor of the failure prediction response, the requestor being the user, and the portion of the one or more structured knowledge attributes that provide for interpretability of the failure prediction further provides visibility for the user into understanding how and why the trained machine learning model generated the failure prediction of the failure prediction response, using the architecture, in a manner that the trained machine learning model generated the failure prediction while the architecture of the trained machine learning model remains hidden from the user. ### Claim 9 (depends on 1) The method of claim 1, wherein the one or more structured knowledge attributes comprise parameters on which the trained machine learning model is trained to generate the inferences. ### Claim 10 (depends on 9) The method of claim 9, wherein the one or more structured knowledge attributes further comprise relationships between components of the trained machine learning model, the components comprising at least input features of data ingested into the trained machine learning model, the inferences generated by the trained machine learning model, and one or more rules followed by the trained machine learning model to generate the inferences. ### Claim 11 (depends on 1) The method of claim 1, wherein the architecture of the trained machine learning model comprises at least information regarding one or more other trained machine learning models that were used to train the trained machine learning model and regarding respective other architectures of each of the one or more other trained machine learning models. ### Claim 12 A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor of a data processing system manager configured to manage data processing systems, cause the data processing system manager to perform operations for managing failures of the data processing systems, the operations comprising: generating a structured knowledge labeled dataset using: one or more structured knowledge attributes stored in a structured knowledge repository, the one or more structured knowledge attributes being extracted from an architecture of a trained machine learning model, the architecture controls how the trained machine learning model generates inferences, and one or more failure prediction prompts stored in a failure prediction prompt library; fine-tuning an inference model using the structured knowledge labeled dataset to obtain a trained inference model; using the trained inference model to generate a failure prediction response, the failure prediction response comprising a failure prediction and a portion of the one or more structured knowledge attributes that provide for interpretability of the failure prediction, the failure prediction being one of the inferences; and providing the failure prediction response to a user wherein prior to generating the structured knowledge labeled dataset, the operations further comprise: extracting, using one or more techniques comprising explainable artificial intelligence (AI) techniques, the one or more structured knowledge attributes from the architecture of trained machine learning model; and storing the one or more structured knowledge attributes extracted from trained machine learning model into the structured knowledge repository. ### Claim 13 (depends on 12) The non-transitory machine-readable medium of claim 12, wherein prior to using the trained inference model to generate the failure prediction response, the operations further comprise: obtaining a failure prediction response request from the user for data stored in the structured knowledge repository, the data comprising the one or more structured knowledge attributes, and the data being usable to manage an indication of failure for a data processing system of the data processing systems. ### Claim 14 (depends on 13) The non-transitory machine-readable medium of claim 13, wherein the one or more failure prediction prompts include at least a short readout prompt, a long readout prompt, and a knowledge-base-article-based prompt. ### Claim 15 (depends on 14) The non-transitory machine-readable medium of claim 14, wherein the inference model is a large language model (LLM), and wherein fine-tuning the inference model using the structured knowledge labeled dataset comprises implementation of a private infrastructure fine-tuning method. ### Claim 16 (depends on 14) The non-transitory machine-readable medium of claim 14, wherein the inference model is a large language model (LLM), and wherein fine-tuning the inference model using the structured knowledge labeled dataset comprises implementation of a public infrastructure fine-tuning method. ### Claim 17 A data processing system manager, comprising: a processor; and a memory coupled to the processor to store instructions, which when executed by the processor, cause the data processing system manager to perform operations for managing failures of data processing systems of the data processing systems, the operations comprising: generating a structured knowledge labeled dataset using: one or more structured knowledge attributes stored in a structured knowledge repository, the one or more structured knowledge attributes being extracted from an architecture of a trained machine learning model, the architecture controls how the trained machine learning model generates inferences, and one or more failure prediction prompts stored in a failure prediction prompt library; fine-tuning an inference model using the structured knowledge labeled dataset to obtain a trained inference model; using the trained inference model to generate a failure prediction response, the failure prediction response comprising a failure prediction and a portion of the one or more structured knowledge attributes that provide for interpretability of the failure prediction, the failure prediction being one of the inferences; and providing the failure prediction response to a user wherein prior to generating the structured knowledge labeled dataset, the operations further comprise: extracting, using one or more techniques comprising explainable artificial intelligence (AI) techniques, the one or more structured knowledge attributes from the architecture of trained machine learning model; and storing the one or more structured knowledge attributes extracted from trained machine learning model into the structured knowledge repository. ### Claim 18 (depends on 17) The data processing system manager of claim 17, wherein prior to using the trained inference model to generate the failure prediction response, the operations further comprise: obtaining a failure prediction response request from the user for data stored in the structured knowledge repository, the data comprising the one or more structured knowledge attributes, and the data being usable to manage an indication of failure for a data processing system of the data processing systems. ### Claim 19 (depends on 18) The data processing system manager of claim 18, wherein the one or more failure prediction prompts include at least a short readout prompt, a long readout prompt, and a knowledge-base-article-based prompt. ### Claim 20 (depends on 17) The data processing system manager of claim 17, wherein the architecture of the trained machine learning model is hidden from a requestor of the failure prediction response, the requestor being the user, and the portion of the one or more structured knowledge attributes that provide for interpretability of the failure prediction further provides visibility for the user into understanding how and why the trained machine learning model generated the failure prediction of the failure prediction response, using the architecture, in a manner that the trained machine learning model generated the failure prediction while the architecture of the trained machine learning model remains hidden from the use.