# US 10,320,954 B2 · Diffusing packets to identify faulty network apparatuses in multipath inter-data center networks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,320,954 B2 - Document ID: US10320954B2 - Status: Granted - Kind: B2 - Application number: 15/424,439 - Filed: 2017-02-03 - Published: 2018-08-09 - Granted: 2019-06-11 - Earliest claimed: 2017-02-03 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Bradley David Rutkowski; Srinivasachakrapani Kotipalli; Mathrubootham Janakiraman; Veenu Singla - Web page: https://patentgrep.com/patent/US10320954B2 ## CPC - H04L 69/161: {Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields} - H04L 41/0677: Localisation of faults - H04L 43/0829: Packet loss - H04L 45/24: Multipath - H04L 69/14: Multichannel or multilink protocols - H04L 45/7453: using hashing - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract The techniques described herein describe diffusing packets through multipath inter-data center networks to identify faulty network apparatuses therein. In an implementation, a method of diffusing packets through a multipath inter-data center network to identify a faulty network apparatus in a source data center is disclosed. The method includes diffusing packets through various paths of a network fabric. Diffusing the packets includes sending out bursts of packets with varying packet header parameters to a set of machines in one or more target data centers. The method further include monitoring packet drops from the bursts of packets, issuing tracerts for each of the packet drops to identify corresponding failed routes, and triangulating the failed routes to identify the faulty network apparatus. ## Claims ### Claim 1 A method of diffusing packets through a multipath inter-data center network to identify a faulty network apparatus in a network fabric of a source data center, the method comprising: identifying, by one or more source machines in the source data center, a set of target machines in one or more target data centers, wherein each target machine of the set of target machines has one or more available destination ports and each source machine of the one or more source machines has one or more available source ports; determining, based on the available source ports and the available destination ports, each available source port/destination port combination between the source data center and the one or more target data centers; diffusing the packets through various paths of the network fabric by sending out bursts of packets with varying packet header parameters to each of the set of target machines including one or more packets for each available source port/destination port combination; monitoring packet drops from the bursts of packets; issuing tracerts for each of the packet drops to identify corresponding failed routes; and triangulating the failed routes to identify the faulty network apparatus. ### Claim 2 (depends on 1) The method of claim 1, wherein the packets comprise transmission control protocol (TCP) packets. ### Claim 3 (depends on 1) The method of claim 1, wherein the packet header parameters comprise a source port and a destination port, and wherein diffusing the packets comprises bursting multiple packets for each available source port/destination port combination. ### Claim 4 (depends on 1) The method of claim 1, wherein the packet header parameters comprise a source port, a destination port, a source IP address, and a destination IP address, and wherein diffusing the packets comprises bursting multiple packets for each available source port/destination port/source IP address/destination IP address combination. ### Claim 5 (depends on 1) The method of claim 1, wherein each tracert indicates a route from a source machine in the source data center through network apparatuses of the network fabric to a target machine in a target data center of the one or more target data centers. ### Claim 6 (depends on 4) The method of claim 4, wherein each tracert further indicates IP addresses and port information. ### Claim 7 (depends on 1) The method of claim 1, wherein packet drops are detected when an acknowledgement indicating safe receipt of a packet is not received within a predetermined period of time. ### Claim 8 (depends on 1) The method of claim 1, wherein triangulating the failed paths to identify the faulty network apparatus comprises: computing a number of times each network apparatus occurs in the failed routes; and identifying the faulty network apparatus as the network apparatus occurring the highest number of times in the failed routes. ### Claim 9 (depends on 8) The method of claim 8, wherein faulty network apparatus comprises a faulty router. ### Claim 10 (depends on 1) The method of claim 1, wherein triangulating the failed paths to identify the faulty network apparatus comprises: computing a number of times each set of neighbor devices occurs in the failed routes, wherein each set of the neighbor devices includes a first device and a second device; identifying a set of the neighbor devices occurring the highest number of times in the failed routes; and identifying the faulty network apparatus as an apparatus connecting the set of the neighbor devices. ### Claim 11 (depends on 10) The method of claim 10, wherein the faulty network apparatus comprises a network link or interface. ### Claim 12 (depends on 1) The method of claim 1, wherein diffusing the packets further comprises: identifying available machines in the one or more target data centers; and adding the available machines to a list containing the set of available machines. ### Claim 13 A data center comprising: a multipath network fabric including a multi-layer set of redundant routers; multiple racks each comprising multiple machines, wherein at least a subset of the machines are configured to: identify a set of target machines in one or more target data centers, wherein each target machine of the set of target machines has one or more available destination ports and each source machine of the one or more source machines has one or more available source ports; determine, based on the available source ports and the available destination ports, each available source port/destination port combination between the source data center and the one or more target data centers; diffuse packets through various paths of the multipath fabric to collectively identify a faulty network apparatus, wherein diffusing the packets comprises sending out bursts of packets with varying packet header parameters to each available source port/destination port combination, issue tracerts for each packet drop from the bursts of packets to identify corresponding failed routes, consolidate the failed routes from the subset of the machines, and triangulate the consolidated failed routes to identify the faulty network apparatus. ### Claim 14 (depends on 13) The data center of claim 13, wherein the subset of the machines are further configured to monitor the packet drops from the burst of packets. ### Claim 15 (depends on 13) The data center of claim 13, wherein the packet header parameters comprise a source port and a destination port, and wherein diffusing the packets comprises bursting multiple packets for each available source port/destination port combination. ### Claim 16 (depends on 13) The data center of claim 13, wherein the faulty network apparatus comprises a failed router of the set of redundant routers, and wherein to triangulate the failed paths, the subset of the machines are further configured to compute a number of times each router of the set of redundant routers occurs in the failed routes. ### Claim 17 (depends on 13) The data center of claim 13, wherein the faulty network apparatus comprises a failed network link or interface, and wherein to triangulate the failed paths, the subset of machines are further configured to compute a number of times each set of neighbor devices occurs in the failed routes. ### Claim 18 An apparatus comprising: one or more computer readable storage media; and program instructions stored on the one or more computer readable storage media which, when executed by one or more processing systems, direct the one or more processing systems to at least: identify a set of target machines in one or more target data centers, wherein each target machine of the set of target machines has one or more available destination ports and each source machine of the one or more source machines has one or more available source ports; determine, based on the available source ports and the available destination ports, each available source port/destination port combination between the source data center and the one or more target data centers; diffuse packets through various paths of a multipath inter-data center network fabric, wherein to diffuse the packets, the one or more processing systems sends out bursts of packets with varying packet header parameters to each available source port/destination port combination; monitor packet drops from the bursts of packets; issue tracerts for each of the packet drops to identify corresponding failed routes; and triangulate the failed routes to identify a faulty network apparatus in a source data center. ### Claim 19 (depends on 18) The apparatus of claim 18, wherein to triangulate the failed paths, the instructions, when executed by the processing system, further direct the processing system to: compute a number of times each network apparatus occurs in the failed routes; and identify the faulty network apparatus as the network apparatus occurring the highest number of times in the failed routes. ### Claim 20 (depends on 18) The apparatus of claim 18, wherein to triangulate the failed paths, the instructions, when executed by the processing system, further direct the processing system to: compute a number of times each set of neighbor devices occurs in the failed routes, wherein each set of the neighbor devices includes a first device and a second device; identify a set of the neighbor devices occurring the highest number of times in the failed routes; and identify the faulty network apparatus as an apparatus connecting the set of the neighbor devices.