# US 10,769,017 B2 · Adaptive multi-level checkpointing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,769,017 B2 - Document ID: US10769017B2 - Status: Granted - Kind: B2 - Application number: 15/960,302 - Filed: 2018-04-23 - Published: 2019-10-24 - Granted: 2020-09-08 - Earliest claimed: 2018-04-23 - Assignees: Hewlett Packard Enterprise Development LP - Applicants: Hewlett Packard Enterprise Development LP - Inventors: Cong Xu; Itir Akgun; Paolo Faraboschi - Web page: https://patentgrep.com/patent/US10769017B2 ## CPC - G06F 11/1407: {Checkpointing the instruction stream} - G06F 11/1438: {Restarting or rejuvenating} - G06F 11/1451: {by selection of backup contents} - G06F 11/1464: {for networked environments} - H04L 67/1095: Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes - H04L 67/1097: for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS] - G06F 2201/81: Threshold - G06F 2201/84: Using snapshots, i.e. a logical point-in-time copy of the data - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract In some examples, with respect to adaptive multi-level checkpointing, a transfer parameter associated with transfer of checkpoint data from a node-local storage to a parallel file system may be ascertained for the checkpoint data stored in the node-local storage. The transfer parameter may be compared to a specified transfer parameter threshold. A determination may be made, based on the comparison of the transfer parameter to the specified transfer parameter threshold, as to whether to transfer the checkpoint data from the node-local storage to the parallel file system. ## Claims ### Claim 1 An apparatus comprising: a processor; and a non-transitory computer readable medium storing machine readable instructions that when executed by the processor cause the processor to: ascertain, for checkpoint data stored in node-local storage, a transfer parameter associated with transfer of the checkpoint data from the node-local storage to a parallel file system; compare the transfer parameter to a specified transfer parameter threshold determine, based on the comparison of the transfer parameter to the specified transfer parameter threshold, whether to transfer the checkpoint data from the node-local storage to the parallel file system, wherein the transfer parameter includes input/output bandwidth or a percentage disk idle time; and where the transfer parameter includes the input/output bandwidth, based on a determination that the input/output bandwidth is less than the specified input/output bandwidth threshold, cause the transfer of the checkpoint data from the node-local storage to the parallel file system; or where the transfer parameter includes the percentage disk idle time, based on a determination that the percentage disk idle time is less than the specified percentage disk idle time threshold, cause the transfer of the checkpoint data from the node-local storage to the parallel file system. ### Claim 2 (depends on 1) The apparatus according to claim 1, wherein the instructions to compare the transfer parameter to the specified transfer parameter threshold are further to cause the processor to: determine a transfer interval associated with the transfer of the checkpoint data from the node-local storage to the parallel file system; and compare, before expiration of the transfer interval associated with the transfer of the checkpoint data from the node-local storage to the parallel file system, the transfer parameter to the specified transfer parameter threshold. ### Claim 3 (depends on 1) The apparatus according to claim 1, wherein the instructions to compare the transfer parameter to the specified transfer parameter threshold are further to cause the processor to: determine a transfer interval associated with the transfer of the checkpoint data from the node-local storage to the parallel file system; and compare, upon expiration of the transfer interval associated with the transfer of the checkpoint data from the node-local storage to the parallel file system, the transfer parameter to the specified transfer parameter threshold. ### Claim 4 (depends on 1) The apparatus according to claim 1, wherein the transfer parameter includes the input/output bandwidth. ### Claim 5 (depends on 1) The apparatus according to claim 1, wherein the instructions to determine, based on the comparison of the input/output bandwidth to the specified input/output bandwidth threshold, whether to transfer the checkpoint data from the node-local storage to the parallel file system are further to cause the processor to: based on a determination that the input/output bandwidth is greater than or equal to the specified input/output bandwidth threshold, delay the transfer of the checkpoint data from the node-local storage to the parallel file system. ### Claim 6 (depends on 1) The apparatus according to claim 1, wherein the transfer parameter includes the percentage disk idle time. ### Claim 7 A computer implemented method comprising: comparing a storage parameter associated with storage of checkpoint data to a specified storage parameter threshold; determining, based on the comparison, whether to store the checkpoint data in a higher order memory tier or in a lower order memory tier, wherein the lower order memory tier is to be used upon occurrence of spillover associated with the higher order memory tier; based on a determination that a value of the storage parameter is greater than or equal to a value of the specified storage parameter threshold, storing the checkpoint data in the lower order memory tier; and determining, based on a comparison of a transfer parameter to a specified transfer parameter threshold, whether to transfer the checkpoint data from a node-local storage of the lower order memory tier to a parallel file system of the lower order memory tier. ### Claim 8 (depends on 7) The computer implemented method according to claim 7, wherein the storage parameter includes memory bandwidth usage, further comprising: based on a determination that the memory bandwidth usage is less than the specified memory bandwidth usage threshold, storing the checkpoint data in the higher order memory tier. ### Claim 9 (depends on 7) The computer implemented method according to claim 7, wherein the higher order memory tier includes a primary persistent memory, and the lower order memory tier includes a node-local storage or a parallel file system. ### Claim 10 (depends on 7) The computer implemented method according to claim 7, further comprising: determining a transfer interval associated with the transfer of the checkpoint data from the node-local storage to the parallel file system; and comparing, upon expiration of the transfer interval associated with the transfer of the checkpoint data from the node-local storage to the parallel file system, the transfer parameter to the specified transfer parameter threshold. ### Claim 11 (depends on 7) The computer implemented method according to claim 7, further comprising: determining a transfer interval associated with the transfer of the checkpoint data from the node-local storage to the parallel file system; and comparing, before expiration of the transfer interval associated with the transfer of the checkpoint data from the node-local storage to the parallel file system, the transfer parameter to the specified transfer parameter threshold. ### Claim 12 A non-transitory computer readable medium having stored thereon machine readable instructions, the machine readable instructions, when executed, cause a processor to: ascertain a storage parameter associated with storage of checkpoint data; compare the storage parameter to a specified storage parameter threshold; and determine, based on the comparison of the storage parameter to the specified storage parameter threshold, whether to store the checkpoint data in a higher order memory tier or in a lower order memory tier, wherein the lower order memory tier is to be used upon occurrence of spillover associated with the higher order memory tier. ### Claim 13 (depends on 12) The non-transitory computer readable medium according to claim 12, wherein the storage parameter includes memory bandwidth usage. ### Claim 14 (depends on 13) The non-transitory computer readable medium according to claim 13, wherein the machine readable instructions, when executed, further cause the processor to: based on a determination that the memory bandwidth usage is less than the specified memory bandwidth usage threshold, store the checkpoint data in the higher order memory tier. ### Claim 15 (depends on 13) The non-transitory computer readable medium according to claim 13, wherein the machine readable instructions, when executed, further cause the processor to: based on a determination that the memory bandwidth usage is greater than or equal to the specified memory bandwidth usage threshold, store the checkpoint data in the lower order memory tier. ### Claim 16 (depends on 12) The non-transitory computer readable medium according to claim 12, wherein the higher order memory tier includes a primary persistent memory, and the lower order memory tier includes a node-local storage or a parallel file system. ### Claim 17 (depends on 13) The non-transitory computer readable medium according to claim 13, wherein the machine readable instructions, when executed, further cause the processor to: based on a determination that the memory bandwidth usage is greater than or equal to the specified memory bandwidth usage threshold, store a part of the checkpoint data in the higher order memory tier according to a capacity of the higher order memory tier, and store a remaining part of the checkpoint data in the lower order memory tier. ### Claim 18 (depends on 13) The non-transitory computer readable medium according to claim 13, wherein the machine readable instructions, when executed, further cause the processor to: based on a determination that the memory bandwidth usage is greater than or equal to the specified memory bandwidth usage threshold, store a part of the checkpoint data in the higher order memory tier up to the specified memory bandwidth usage threshold, and store a remaining part of the checkpoint data in the lower order memory tier.