# US 10,372,647 B2 · Exascale fabric time synchronization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,372,647 B2
- Document ID: US10372647B2
- Status: Granted
- Kind: B2
- Application number: 14/977,773
- Filed: 2015-12-22
- Published: 2017-06-22
- Granted: 2019-08-06
- Earliest claimed: 2015-12-22
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Thomas D. Lovett; Michael A. Parker; Mark S. Birrittella
- Web page: https://patentgrep.com/patent/US10372647B2

## CPC

- G06F 13/4022: {using switching circuits, e.g. switching matrix, connection or expansion network (G06F13/4009 takes precedence)}
- G06F 1/12: Synchronisation of different clock signals {provided by a plurality of clock generators}
- G06F 13/364: using independent requests or grants, e.g. using separated request and grant lines
- G06F 13/4282: {on a serial bus, e.g. I2C bus, SPI bus (on daisy chain buses G06F13/4247)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods and apparatus for implementing time synchronization across exascale fabrics. A master clock node is coupled to a plurality of slave nodes via a fabric comprising a plurality of fabric switches and a plurality of fabric links, wherein each slave node is connected to the master clock node via a respective clock tree path that traverses at least one fabric switch. The fabric switches are configured to selectively forward master clock time data internally along paths with fixed latencies that bypass the switches' buffers and switch circuitry, which enables the entire clock tree paths to also have fixed latencies. The fixed latency of the clock tree path is determined for each slave node. The local clocks of the slave nodes are then synchronized with the master clock by using master clock time data received by each slave node and the fixed latency of the clock tree path from the master clock node to the slave node that is determined. Techniques for determining a clock rate mismatch between the master clock and a local clock is also provided.

## Claims

### Claim 1

A method performed in an environment including a master clock node communicatively coupled to a plurality of slave nodes via a fabric comprising a plurality of fabric switches and a plurality of fabric links, comprising:
forwarding master clock time data generated by a master clock operating on the master clock node to each of the plurality of slave nodes via clock tree paths having fixed latencies;
for a given slave node, determining the fixed latency of the clock tree path from the master clock node to the slave node; and
synchronizing a local clock for the slave node and the master clock operating on the master clock node using the master clock time data received by the slave node and the fixed latency of the clock tree path from the master clock node to the slave node that is determined,
wherein, for at least a portion of the slave nodes, the clock tree paths from the master clock node to those slave nodes traverse multiple fabric switches.

### Claim 2 (depends on 1)

The method of claim 1, wherein each of the multiple fabric switches that are traversed for a given forwarding path between the master clock node and a slave node comprises:
a plurality of ports, each including a receive port operatively coupled to a receive buffer and a transmit port operative coupled to a transmit buffer;
switching circuitry that, during operation, enables selectable connection between a receive buffer of a first port and a transmit buffer of a second port;
at least one bypass circuit path between a receive port of an upstream clock port and a transmit port of a downstream clock port that enables data received at the receive port to be forwarded to the transmit port along a bypass circuit path.

### Claim 3 (depends on 2)

The method of claim 2, wherein the at least one circuit path comprises a ring bus to which at least a portion of the receive ports and transmit ports are operatively coupled.

### Claim 4 (depends on 2)

The method of claim 2, wherein the at least one circuit path comprises first and second ring buses to which at least a portion of the receive ports and transmit ports are operatively coupled, wherein data is forwarded in a clockwise direction for the first ring bus and data is forwarded in a counterclockwise direction for the second ring bus, and wherein the data is forwarded along one of the first and second ring buses based on which ring bus results in the least latency.

### Claim 5 (depends on 1)

The method of claim 1, wherein a clock tree path between a master clock node and a slave node traverses a plurality of fabric switches and a plurality of link segments linking the plurality of fabric switches in communication, and where determining the fixed latency of a clock tree path comprises:
determining a latency for each of the plurality of link segments;
determining a switch forwarding latency from an upstream clock port to a downstream clock port within each of the plurality of fabric switches, wherein the clock tree path includes a first link segment coupled to the upstream clock port and a second link segment coupled to the downstream clock port; and
aggregating the latencies for each of the plurality of link segments and the switch forwarding latencies.

### Claim 6 (depends on 1)

The method of claim 1, wherein data is transferred across the plurality of fabric links using a stream of flow control units (flits), and wherein the master clock time data is contained within at least one flit that is forwarded along the clock tree paths from the master clock node to the plurality of slave nodes.

### Claim 7 (depends on 6)

The method of claim 6, wherein the master clock time data is contained in first and second flits, the first flit comprising an upper bit portion of the master clock time data and the second flit containing a lower bit portion of the master clock time data.

### Claim 8 (depends on 7)

The method of claim 7, wherein a portion of the upper bit portion of the master clock time data in the first flit and the lower bit portion of the master clock time data in the second flit are overlapping, further comprising comparing the master clock time data bits in overlapping portion to determine that the upper bit portion and lower bit portion of the master clock time data is consistent.

### Claim 9 (depends on 6)

The method of claim 6, further comprising:
receiving a flit stream comprising a stream of flits at a receive port of an upstream clock port of a fabric switch, the receive port having an associated receive buffer;
detecting that a first plurality of flits in the flit stream contain data corresponding to fabric packets;
storing the first plurality of flits in a receive buffer for the receive port;
detecting that a flit in the flit stream corresponds to a Time control flit containing master clock time data;
extracting the master clock time data from the Time control flit;
forwarding the master clock time data from the receive port to a downstream clock port in the fabric switch along a bypass circuit path;
encapsulating the master clock time data in a new Time control flit generated at a transmit port of the downstream clock port; and
injecting the new Time control flit into a flit stream that is sent outbound onto a fabric link from the transmit port.

### Claim 10 (depends on 6)

The method of claim 6, further comprising:
receiving a flit stream comprising a stream of flits at a receive port of an upstream clock port of a fabric switch, the receive port having an associated receive buffer;
detecting that a flit in the flit stream corresponds to a Time control flit containing master clock time data;
forwarding the Time control flit from the receive port to a downstream clock port in the fabric switch without buffering the Time control flit in the receive buffer associated with the receive port; and
injecting the Time control flit into a flit stream that is sent outbound onto a fabric link from a transmit port of the downstream clock port.

### Claim 11 (depends on 6)

The method of claim 6, wherein, for each slave node, the at least one flit containing the master clock time data is forwarded along a clock tree path from the master clock node to the slave node in a manner that guarantees data integrity of the master clock time data forwarded to the slave node.

### Claim 12 (depends on 1)

The method of claim 1, further comprising determining a clock rate mismatch between the master clock and a local clock on a slave node.

### Claim 13 (depends on 12)

The method of claim 12, wherein determining the clock rate mismatch between the master clock and the local clock comprises:
receiving first master clock time data at the slave node;
storing a first master clock time sample corresponding to the first master clock time data at the slave node;
in conjunction with receiving the first master clock time data, storing a first sample of the local clock at the slave node;
receiving second master clock time data at the slave node;
storing the second master clock time data at the slave node;
in conjunction with receiving the second master clock time data, storing a second sample of the local clock at the slave node;
subtracting the first master clock time data from the second master clock time data to obtain a master clock time sample difference;
subtracting the first sample from the second sample of the local clock to obtain a time difference for the local clock time sample difference; and
subtracting the local clock time sample difference from the master clock time sample difference and dividing the result by the local clock time sample difference to obtain the clock rate mismatch.

### Claim 14 (depends on 1)

The method of claim 1, further comprising:
replacing a master clock node with a new master clock node; and
forwarding master clock time data generated by a master clock operating on the new master clock node to each of the plurality of slave nodes via clock tree paths having fixed latencies;
for a given slave node, determining the fixed latency of the clock tree path from the new master clock node to the slave node; and
synchronizing a local clock for the slave node and the master clock at the new master clock node using the master clock time data received by the slave node and the fixed latency of the clock tree path from the master clock node to the slave node that is determined.

### Claim 15

A fabric switch, comprising:
a plurality of ports, each including a receive port operatively coupled to a receive buffer and a transmit port operative coupled to a transmit buffer, at least one port comprising an upstream clock port and at least one port comprising a downstream clock port;
switching circuitry that, during operation, enables selectable connection between a receive buffer of a first port and a transmit buffer of a second port;
at least one bypass circuit path between a receive port of an upstream clock port and a transmit port of a downstream clock port that enables master clock time data received at the receive port of the upstream clock port to be forwarded to the transmit port of the downstream clock port in a manner that bypasses the receive buffer for the receive port, and the transmit buffer for the transmit port,
wherein the receive port of the upstream clock port is configured to extract master clock time data when it is received at the receive port.

### Claim 16 (depends on 15)

The fabric switch of claim 15, wherein the at least one circuit path comprises at least one ring bus having a plurality of ring node stops including respective ring node stops operatively coupled to each of the at least one upstream clock port and each of the at least one downstream clock port.

### Claim 17 (depends on 16)

The fabric switch of claim 16, wherein the fabric switch is configured to be implemented in a fabric that transfers data along fabric links in a stream of flow control units (flits) comprising flit streams, and the fabric switch is further configured to:
receive a flit stream at an upstream clock port;
detect a Time control flit in the flit stream;
extract master clock time data from the Time control flit;
forward the master clock time data along a first ring bus; and
at each of a multiple ring stop nodes on the first interconnect;
encapsulate a copy of the master clock time data in a new Time control flit; and
inject the new Time control flit into a flit stream sent outbound onto a fabric link from the downstream clock port to which the ring stop node is coupled.

### Claim 18 (depends on 16)

The fabric switch of claim 16, wherein the fabric switch is configured to be implemented in a fabric that transfers data along fabric links in a stream of flow control units (flits) comprising flit streams, and the fabric switch is further configured to:
receive a flit stream at an upstream clock port;
detect a Time control flit in the flit stream;
forward the Time control flit along a first ring bus; and
at each ring stop node on the first interconnect;
inject a copy of the Time control flit into a flit stream sent outbound onto a fabric link from the downstream clock port to which the ring stop node is coupled.

### Claim 19 (depends on 15)

The fabric switch of claim 15, wherein the at least one circuit path comprises first and second ring busses, each having a plurality of ring node stops including respective ring node stops operatively coupled to each of the at least one upstream clock port and each of the at least one downstream clock port, wherein data is forwarded along the first ring bus in a clockwise direction and data is forwarded along a second ring bus in a counter-clockwise direction.

### Claim 20 (depends on 15)

The fabric switch of claim 15, wherein the fabric switch is configured to be implemented in a fabric that transfers data along fabric links in a stream of flow control units (flits) comprising flit streams, the fabric switch further comprising:
Time control flit extraction logic at each of the at least one upstream clock port, configured to extract Time control flits from a received flit stream comprising master clock time data and forward one of a) an extracted Time control flit or b) master time data contained in an extracted Time control flit along a bypass circuit path between the upstream clock port and at least one downstream clock port.

### Claim 21 (depends on 20)

The fabric switch of claim 20, further comprising:
Time control flit injection logic at each of the at least one downstream clock ports, configured to at least one of,
a) receive master time data forwarded from an upstream clock port along the bypass circuit path, generate a new Time control flit, and inject the new Time control flit into a flit stream sent outbound onto a fabric link from the downstream clock port; and
b) receive a Time control flit forwarded from an upstream clock port along the bypass circuit path, and inject the Time control flit into a flit stream sent outbound onto a fabric link from the downstream clock port.

### Claim 22

A system comprising:
a master clock node having a master clock and including a host fabric interface (HFI);
a plurality of slave nodes, each including a local clock and including an HFI;
a plurality of fabric switches, each including,
a plurality of ports, each including a receive port operatively coupled to a receive buffer and a transmit port operative coupled to a transmit buffer, at least one port comprising an upstream clock port and at least one port comprising a downstream clock port;
switching circuitry that, during operation, enables selectable connection between pairs of receive buffers and transmit buffers of different ports;
at least one bypass circuit path between the at least one upstream clock port and the at least one downstream clock port; and
a plurality of fabric links interconnect ports on the plurality of fabric switches, the master clock node, and the plurality of slave nodes,
wherein during operation of the system the master clock node generates master clock time data that is forwarded to each of the slave nodes via a clock tree path having a fixed latency, wherein each clock tree path traverses the bypass circuit path for at least one fabric switch and a plurality of fabric links.

### Claim 23 (depends on 22)

The system of claim 22, wherein each fabric switch is configured to forward master clock time data along the at least one bypass circuit path in a manner that bypasses the receive buffers, the transmit buffers, and the switching circuitry.

### Claim 24 (depends on 23)

The system of claim 23, wherein the system performs operations to facilitate synchronization of the local clocks on the slave nodes with the master clock comprising:
forwarding master clock time data generated by the master clock to each of the plurality of slave nodes via a respective clock tree path having fixed latency;
for each slave node, determining the fixed latency of the clock tree path from the master clock node to the slave node; and
synchronizing a local clock for the slave node and the master clock operating on the master clock node using the master clock time data received by the slave node and the fixed latency of the clock tree path from the master clock node to the slave node that is determined.

### Claim 25 (depends on 23)

The system of claim 23, wherein the system is further configured to:
determine a clock rate mismatch of the local clock on each slave node relative to the master clock; and
employ the clock rate mismatch determined for each local clock to maintain synchronization between the master clock and each local clock.