# US 8,788,879 B2 · Non-volatile memory for checkpoint storage > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,788,879 B2 - Document ID: US8788879B2 - Status: Granted - Kind: B2 - Application number: 13/004,005 - Filed: 2011-01-10 - Published: 2011-07-14 - Granted: 2014-07-22 - Earliest claimed: 2010-01-08 - Assignees: International Business Machines Corporation - Applicants: Matthias A. Blumrich; Dong Chen; Thomas M. Cipolla; Paul W. Coteus; Alan Gara; Philip Heidelberger; Mark J. Jeanson; Gerard V. Kopcsay; Martin Ohmacht; Todd E. Takken - Inventors: Matthias A. Blumrich; Dong Chen; Thomas M. Cipolla; Paul W. Coteus; Alan Gara; Philip Heidelberger; Mark J. Jeanson; Gerard V. Kopcsay; Martin Ohmacht; Todd E. Takken - Web page: https://patentgrep.com/patent/US8788879B2 ## CPC - G06F 11/1438: {Restarting or rejuvenating} - G06F 2201/82: Solving problems relating to consistency - G06F 2201/84: Using snapshots, i.e. a logical point-in-time copy of the data - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A system, method and computer program product for supporting system initiated checkpoints in high performance parallel computing systems and storing of checkpoint data to a non-volatile memory storage device. The system and method generates selective control signals to perform checkpointing of system related data in presence of messaging activity associated with a user application running at the node. The checkpointing is initiated by the system such that checkpoint data of a plurality of network nodes may be obtained even in the presence of user applications running on highly parallel computers that include ongoing user messaging activity. In one embodiment, the non-volatile memory is a pluggable flash memory card. ## Claims ### Claim 1 A system for checkpointing messages in a parallel computing system having a plurality of nodes connected as a network, each node having multiple processors and an associated memory operatively connected therewith via an interconnect device, said checkpointing system comprising: at each node: a non-volatile memory storage device; one or more control units, each control unit adapted to receive command instructions from a processor for controlling data flow of packets received by a network and flow of packets to be transmitted to the network, each said one or more control units coupled to each of a plurality of devices within said node involved with processing of received and transmitted packets for communicating data therebetween; said control unit responsive to a control signal for performing a checkpoint at said node, wherein said control unit generates a control signal to initiate stopping of a flow of packets received by a network and flow of packets to be transmitted to the network; and, a logic device associated with each said plurality of devices and each responsive to a control signal to initiate obtaining said checkpoint data when said packet data flow has stopped, said checkpointing data obtained from said plurality of devices for receipt in register devices associated with each said one or more control units, said control unit responsive to a further control signal for writing out said checkpoint data received at said associated register devices to said non-volatile memory storage device, wherein each said control unit generates selective control signals to perform said checkpointing of system related data in presence of messaging activity associated with a user application running at the node, and each control unit responding to a further control signal for restoring checkpoint data received at said non-volatile memory storage device back to said associated register devices. ### Claim 2 (depends on 1) The system as claimed in claim 1, wherein one node of said plurality of nodes of said network performs is configured as a host node for initiating checkpoint and coordinating reading and writing of said data when performing said checkpoint of each said plurality of nodes in said network. ### Claim 3 (depends on 2) The system as claimed in claim 2, wherein said plurality of devices includes one or more of a network injection FIFO buffers for receiving packets to be injected into a network via a corresponding sender device, and, one or more network reception FIFO buffers for receiving packets via a corresponding receiver device from the network. ### Claim 4 (depends on 2) The system as claimed in claim 2, wherein a logic device includes a finite state machine controlling operation of said network injection FIFO buffers and said corresponding sender device and network reception FIFO buffers and corresponding receiver device, said control signal comprising bits, a separate bit associated with a respective network injection FIFO buffer, sender device, reception FIFO buffer and receiver device, said control unit adapted to set said bits and input said bits to the finite state machine for ensuring completing any packet in progress and preventing processing on subsequent packets. ### Claim 5 (depends on 2) The system as claimed in claim 2, wherein said plurality of devices includes: a plurality of receive Direct Memory Access (DMA) engine devices, each configured to store received message packets in a memory associated with a processor device from a network reception FIFO buffer; and, a plurality of injection DMA engine devices, each configured to store message packet data from a memory associated with a processor device to a network injection FIFO buffer, wherein a logic device comprises a finite state machine implemented at each respective said plurality of injection DMA engine devices and each respective said plurality of receive DMA engine devices, for obtaining checkpoint of state data associated with processing of system and user message packets. ### Claim 6 (depends on 2) The system as claimed in claim 2, wherein each node further comprises means for detecting when said packet data flow has stopped at said node, said host node further comprising: means for setting up a first barrier, such that all nodes in said network having detected packet data flow stoppage, call said first barrier prior to initiating said obtaining checkpoint data at each respective node. ### Claim 7 (depends on 6) The system as claimed in claim 6, wherein each node further comprises means for detecting completion of obtaining and writing out said checkpoint data to said memory storage device, said host node further comprising: means for setting up a second barrier, such that all nodes in said network having detected completion of writing checkpointing data to said memory storage, call said second barrier prior to restoring said checkpoint data back to said plurality of devices at each respective node. ### Claim 8 (depends on 7) The system as claimed in claim 7, wherein each node further comprises means for detecting completion of restoring said checkpoint data back to said plurality of devices at each respective node, said host node further comprising: means for setting up a third barrier, such that all nodes in said network having detected completion of restoring checkpointing data to said plurality of devices, call said third barrier prior to resuming system and user messaging at each respective said plurality of nodes. ### Claim 9 (depends on 1) The system as claimed in claim 1, wherein an external network performs as a host node for initiating checkpoint and coordinating reading and writing of said data when performing said checkpoint of each said plurality of nodes in said network. ### Claim 10 (depends on 1) The system as claimed in claim 1, wherein said non-volatile memory storage device is a pluggable flash memory card. ### Claim 11 A method for checkpointing messages in a parallel computing system having a plurality of nodes connected as a network, each node having multiple processor units and an associated memory operatively connected therewith via an interconnect device, said method comprising: receiving, at one or more control units, a command instruction from a processor for controlling data flow of packets received by a network and flow of packets to be transmitted to the network, each said one or more control units coupled to each of a plurality of devices within said node involved with processing of received and transmitted packets for communicating data therebetween; performing, at each said node, a checkpoint, said performing including: generating, at said control units, a control signal to initiate stopping flow of packets received by a network and flow of packets to be transmitted to the network; and, responding to a first control signal received at a logic device associated with each said plurality of devices, to initiate obtaining said checkpoint data when said packet data flow has stopped, said checkpointing data obtained from said plurality of devices for receipt in register devices associated with each said one or more control units; and, responding to a second control signal for writing out said checkpoint data received at said associated register devices to a non-volatile memory storage device, wherein each said control unit generates selective control signals to perform said checkpointing of system related data in presence of messaging activity associated with a user application running at the node, and each control unit responding to a further control signal for restoring checkpoint data received at said non-volatile memory storage device back to said associated register devices. ### Claim 12 (depends on 11) The method as claimed in claim 11, wherein one node of said plurality of nodes of said network is configured as a host node for initiating checkpoint and coordinating reading and writing of said data when performing said checkpoint of each said plurality of nodes in said network. ### Claim 13 (depends on 12) The method as claimed in claim 12, wherein said plurality of devices includes one or more of a network injection FIFO buffers for receiving packets to be injected into a network via a corresponding sender device, and, one or more network reception FIFO buffers for receiving packets via a corresponding receiver device from the network. ### Claim 14 (depends on 12) The method as claimed in claim 12, wherein a logic device includes a finite state machine controlling operation of said network injection FIFO buffers and said corresponding sender device and network reception FIFO buffers and corresponding receiver device, said control signal comprising bits, a separate bit associated with a respective network injection FIFO buffer, sender device, reception FIFO buffer and receiver device, said method further comprising: setting, at said control unit, said bits and inputting said bits to the finite state machine for ensuring completing any packet in progress and preventing processing on subsequent packets. ### Claim 15 (depends on 12) The method as claimed in claim 12, wherein each said plurality of devices further includes: a plurality of receive Direct Memory Access (DMA) engine devices, each configured to store received message packets in a memory associated with a processor device from a network reception FIFO buffer; and, a plurality of injection DMA engine devices, each configured to store message packet data from a memory associated with a processor device to a network injection FIFO buffer, wherein a logic device comprises a finite state machine implemented at each respective said plurality of injection DMA engine devices and each respective said plurality of receive DMA engine devices, said method comprising: obtaining, from a logic device, respective checkpoint of state data associated with processing of system and user message packets. ### Claim 16 (depends on 12) The method as claimed in claim 12, further comprising: at each node, detecting when said packet data flow has stopped at said node; and, setting up a first barrier, such that all nodes in said network having detected packet flow stoppage, call said first barrier prior to initiating said obtaining checkpoint data at each respective node. ### Claim 17 (depends on 16) The method as claimed in claim 16, further comprising: at each node, detecting completion of obtaining and writing out said checkpoint data to said memory storage device, and at said host node: setting up a second barrier, such that all nodes in said network having detected completion of writing checkpointing data to said memory storage, call said second barrier prior to restoring said checkpoint data back to said plurality of devices at each respective node. ### Claim 18 (depends on 17) The method as claimed in claim 17, further comprising: at each node, detecting completion of restoring said checkpoint data back to said plurality of devices at each respective node, and at said host node: setting up a third barrier, such that all nodes in said network having detected completion of restoring checkpointing data to said plurality of devices, call said third barrier prior to resuming system and user messaging at each respective said plurality of nodes. ### Claim 19 (depends on 11) The method as claimed in claim 11, wherein an external network is configured as a host node for initiating checkpoint and coordinating reading and writing of said data when performing said checkpoint of each said plurality of nodes in said network. ### Claim 20 (depends on 11) The method as claimed in claim 11, wherein said non-volatile memory storage device is a pluggable flash memory card. ### Claim 21 A computer program product for checkpointing messages in a parallel computing system having a plurality of nodes connected as a network, each node having multiple processor units and an associated memory operatively connected therewith via an interconnect device, said computer program product comprising: a storage medium, said storage medium not a propagating signal, said storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method comprising: receiving, at one or more control units, a command instruction from a processor for controlling data flow of packets received by a network and flow of packets to be transmitted to the network, each said one or more control units coupled to each of a plurality of devices within said node involved with processing of received and transmitted packets for communicating data therebetween; performing, at each said node, a checkpoint, said performing including: generating, at said control units, a control signal to initiate stopping flow of packets received by a network and flow of packets to be transmitted to the network; and, responding to a first control signal received at a logic device associated with each said plurality of devices, to initiate obtaining said checkpoint data when said packet data flow has stopped, said checkpointing data obtained from said plurality of devices for receipt in register devices associated with each said one or more control units; and, responding to a second control signal for writing out said checkpoint data received at said associated register devices to a non-volatile memory storage device, wherein each said control unit generates selective control signals to perform said checkpointing of system related data in presence of messaging activity associated with a user application running at the node, and each control unit responding to a further control signal for restoring checkpoint data received at said non-volatile memory storage device back to said associated register devices. ### Claim 22 (depends on 21) The computer program product as claimed in claim 21, wherein said non-volatile memory storage device is a pluggable flash memory card.