# US 12,602,302 B2 · Machine learning model training with hardware error simulation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,302 B2 - Document ID: US12602302B2 - Status: Granted - Kind: B2 - Application number: 18/804,236 - Filed: 2024-08-14 - Published: 2026-02-19 - Granted: 2026-04-14 - Earliest claimed: 2024-08-14 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Shirish Bahirat - Web page: https://patentgrep.com/patent/US12602302B2 ## CPC - G06F 11/261: {by simulating additional hardware, e.g. fault simulation} - G06N 20/00: Machine learning - G06N 3/045: Combinations of networks - G06N 3/098: Distributed learning, e.g. federated learning - G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 20/10: using kernel methods, e.g. support vector machines [SVM] - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Methods, systems, and apparatus, including computer-readable storage media for handling component error of datacenter infrastructure by training machine learning models to handle errors when the models are deployed. Rather than predict or mitigate errors in the hardware of a datacenter or another site deploying the machine learning model, the model is trained to perform a task with comparable accuracy and efficiency even when some hardware on which the model is deployed fails. Component error can instead be simulated during training to cause the machine learning model deployed on the infrastructure to learn to correct errors caused by the component errors. The model can continue to be trained to compensate for periods in which not all parts of the model are available at inference. Updates to the model can be backpropagated to correct errors for handling instances of component error. ## Claims ### Claim 1 A method for training a machine learning model, comprising: receiving, by a plurality of hardware components of a computing infrastructure configured to train machine learning models, error frequency data comprising a frequency and duration for component error of the plurality of hardware components; training, by the plurality of hardware components, a machine learning model over one or more training iterations to generate a trained machine learning model, wherein, during at least one of the one or more the training iterations, error of one or more hardware components of the plurality of hardware components is simulated in accordance with the error frequency data; and outputting, by the plurality of hardware components, the trained machine learning model. ### Claim 2 (depends on 1) The method of claim 1, wherein error of the one or more hardware components comprises error of one or more software components implemented by the one or more hardware components. ### Claim 3 (depends on 1) The method of claim 1, wherein training the machine learning model comprises: determining, by the plurality of hardware components, a first model output before simulating error of the one or more hardware components and a second model output after simulating error of the one or more hardware components, calculating, by the plurality of hardware components, a loss between the first model output and the second model output, and updating, by the plurality of hardware components, one or more model parameter values of the machine learning model in accordance with the loss. ### Claim 4 (depends on 1) The method of claim 1, further comprising simulating the error of the one or more hardware components by causing the one or more hardware components to be unresponsive during the training of the machine learning model. ### Claim 5 (depends on 1) The method of claim 1, wherein simulating the error of the one or more hardware components comprises causing the one or more hardware components to lose intermediate data generated during the training of the machine learning model. ### Claim 6 (depends on 1) The method of claim 1, wherein simulating the error of the one or more hardware components comprises simulating at least one of a memory error, a data transmission error, a computation error, or a network connectivity error at the one or more hardware components during the one or more training iterations. ### Claim 7 (depends on 1) The method of claim 1, wherein receiving the error frequency data comprises: receiving historical error data of the plurality of hardware components; and generating, by the plurality of hardware components, the error frequency data based on the historical error data. ### Claim 8 (depends on 7) The method of claim 7, wherein the historical error data comprises a mean-time-between-failure (MTBF) or mean-time-to-first-failure (MTFF) for at least one hardware component of the plurality of hardware components. ### Claim 9 (depends on 1) The method of claim 1, wherein the machine learning model is a foundation model. ### Claim 10 (depends on 1) The method of claim 1, wherein the hardware components comprise one or more processors, one or more memory devices, and one or more network devices for communicating data among hardware components of the plurality of hardware components. ### Claim 11 (depends on 1) The method of claim 1, wherein training the machine learning model comprises: implementing one or more of model, data, or pipeline parallelism; and simulating, by the plurality of hardware components, error based on the implemented one or more parallelisms. ### Claim 12 A system comprising: a plurality of hardware components configured to train machine learning models, and further configured to: receive error frequency data comprising a frequency and duration for component error of the plurality of hardware components; train a machine learning model over one or more training iterations to generate a trained machine learning model, wherein, during one or more of the training iterations, error of one or more hardware components of the plurality of hardware components is simulated in accordance with the error frequency data; and output the trained machine learning model. ### Claim 13 (depends on 12) The system of claim 12, wherein error of the one or more hardware components comprises error of one or more software components implemented by the one or more hardware components. ### Claim 14 (depends on 12) The system of claim 12, wherein in training the machine learning model, the plurality of hardware components are configured to: determine a first model output before simulating error of the one or more hardware components and a second model output after simulating error of the one or more hardware components, calculate a loss between the first model output and the second model output, and update one or more model parameter values of the machine learning model in accordance with the loss. ### Claim 15 (depends on 14) The system of claim 14, wherein the plurality of hardware components is further configured to simulate the error of the one or more hardware components the plurality of hardware components, comprising causing the one or more hardware components to be unresponsive during the training of the machine learning model. ### Claim 16 (depends on 15) The system of claim 15, wherein in simulating the error of the one or more hardware components, the plurality of hardware components is configured to cause the one or more hardware components to lose intermediate data generated during the training of the machine learning model. ### Claim 17 (depends on 15) The system of claim 15, wherein in simulating the error of the one or more hardware components, the plurality of hardware components is configured to simulate at least one of a memory error, a data transmission error, a computation error, or a network connectivity error at the one or more hardware components during the one or more training iterations. ### Claim 18 (depends on 12) The system of claim 12, wherein in receiving the error frequency data, the plurality of hardware components is further configured to: receive historical error data of the plurality of hardware components; and generate the error frequency data based on the historical error data. ### Claim 19 (depends on 18) The system of claim 18, wherein the historical error data comprises a mean-time-to-error (MTBF) for at least one hardware component of the plurality of hardware components. ### Claim 20 One or more non-transitory computer-readable storage media storing instructions that when performed by a plurality of hardware components of a computing infrastructure, causes the plurality of hardware components to perform operations comprising: receiving error frequency data comprising a frequency and duration for component error of the plurality of hardware components; training a machine learning model over one or more training iterations to generate a trained machine learning model, wherein, during one or more of the training iterations, error of one or more hardware components of the plurality of hardware components is simulated in accordance with the error frequency data; and outputting the trained machine learning model.