US 10,372,647 B2 · Exascale fabric time synchronization

Abstract

Methods and apparatus for implementing time synchronization across exascale fabrics. A master clock node is coupled to a plurality of slave nodes via a fabric comprising a plurality of fabric switches and a plurality of fabric links, wherein each slave node is connected to the master clock node via a respective clock tree path that traverses at least one fabric switch. The fabric switches are configured to selectively forward master clock time data internally along paths with fixed latencies that bypass the switches' buffers and switch circuitry, which enables the entire clock tree paths to also have fixed latencies. The fixed latency of the clock tree path is determined for each slave node. The local clocks of the slave nodes are then synchronized with the master clock by using master clock time data received by each slave node and the fixed latency of the clock tree path from the master clock node to the slave node that is determined. Techniques for determining a clock rate mismatch between the master clock and a local clock is also provided.

Claims25

Dependent claims

A method performed in an environment including a master clock node communicatively coupled to a plurality of slave nodes via a fabric comprising a plurality of fabric switches and a plurality of fabric links, comprising:

forwarding master clock time data generated by a master clock operating on the master clock node to each of the plurality of slave nodes via clock tree paths having fixed latencies;
for a given slave node, determining the fixed latency of the clock tree path from the master clock node to the slave node;
and
synchronizing a local clock for the slave node and the master clock operating on the master clock node using the master clock time data received by the slave node and the fixed latency of the clock tree path from the master clock node to the slave node that is determined,
wherein, for at least a portion of the slave nodes, the clock tree paths from the master clock node to those slave nodes traverse multiple fabric switches.

2 more elements

A fabric switch, comprising:

a plurality of ports, each including a receive port operatively coupled to a receive buffer and a transmit port operative coupled to a transmit buffer, at least one port comprising an upstream clock port and at least one port comprising a downstream clock port;
switching circuitry that, during operation, enables selectable connection between a receive buffer of a first port and a transmit buffer of a second port;
at least one bypass circuit path between a receive port of an upstream clock port and a transmit port of a downstream clock port that enables master clock time data received at the receive port of the upstream clock port to be forwarded to the transmit port of the downstream clock port in a manner that bypasses the receive buffer for the receive port, and the transmit buffer for the transmit port,
wherein the receive port of the upstream clock port is configured to extract master clock time data when it is received at the receive port.

1 more element

A system comprising:

a master clock node having a master clock and including a host fabric interface (HFI);
a plurality of slave nodes, each including a local clock and including an HFI;
a plurality of fabric switches, each including,
a plurality of ports, each including a receive port operatively coupled to a receive buffer and a transmit port operative coupled to a transmit buffer, at least one port comprising an upstream clock port and at least one port comprising a downstream clock port;
switching circuitry that, during operation, enables selectable connection between pairs of receive buffers and transmit buffers of different ports;
at least one bypass circuit path between the at least one upstream clock port and the at least one downstream clock port;
and
a plurality of fabric links interconnect ports on the plurality of fabric switches, the master clock node, and the plurality of slave nodes,
wherein during operation of the system the master clock node generates master clock time data that is forwarded to each of the slave nodes via a clock tree path having a fixed latency, wherein each clock tree path traverses the bypass circuit path for at least one fabric switch and a plurality of fabric links.

6 more elements

Dependencies are inferred from the claim text.

Exascale fabric time synchronization

Life of this patent

Exascale fabric time synchronization

Life of this patent

Abstract

Claims25