Failure recovery in a distributed coarse grained reconfigurable architecture

A computing system comprising:

• a host computer;
• a first CGRP (coarse-grained reconfigurable architecture processor) coupled to the host computer through a first external communication link;
• a second CGRP coupled to the host computer through a second external communication link;
• the first CGRP and the second CGRP both respectively including:
• an internal network;
• a first external interface circuit coupled between the host computer and the internal network;
• a second external interface circuit coupled between a third external communication link and the internal network;
• a memory interface circuit coupled between an external memory and the internal network;
• a first array of configurable units and a second array of configurable units, each coupled to the internal network and respectively comprising a plurality of processor configurable units and a plurality of memory configurable units coupled together with an array-level network;
• a first hang detection circuit and a first force-quit controller associated with the first array of configurable units;
• a second hang detection circuit and a second force-quit controller associated with the second array of configurable units;
• and
• a network recovery circuit including one or more control registers that have a first application ID field associated with the first array of configurable units, a second application ID field associated with the second array of configurable units, a first hang syndrome field that indicate whether the first array of configurable units is hung, and a second hang syndrome field that indicate whether the second array of configurable units is hung;
• the host computer programmed to:
• send first configuration information for a first portion of a first application to the first array of configurable units in the first CGRP and set the first application ID field in the first CGRP to a value indicative of the first application;
• send second configuration information for a second portion of the first application to the first array of configurable units in the second CGRP;
• initiate execution of the first application using at least the first array of configurable units in both the first CGRP and the second CGRP;
• and
• in response to receiving an interrupt from the first CGRP to the host computer indicating that the first array of configurable units in the first CGRP is hung, send a command to the second CGRP to send a force quit message to the first force-quit controller to halt execution of the first application on the first array of configurable units in the second CGRP;
• and
• the network recovery circuit in the first CGRP, in response to receiving a hang message from the first hang detection circuit in the first CGRP over the internal network in the first CGRP, is configured to:
• send a first force-quit message to the first force-quit controller in the first CGRP to halt execution by the first array of configurable units in the first CGRP;
• set the first hang syndrome field to indicate that the first array of configurable units in the first CGRP is hung;
• send the interrupt to the host computer to indicate that the first array of configurable units in the first CGRP is hung;
• compare a first value of the first application ID field in the first CGRP to a second value of the second application ID field in the first CGRP;
• and
• send a second force-quit message to the second force-quit controller in the first CGRP to halt execution by the second array of configurable units in the first CGRP in response to the first value being equal to the second value.