Providing a data set for tracking and diagnosing datacenter issues

The system of claim 4 further comprising:

• an analytic controller operable to aggregate log files from the multiple end hosts in the datacenter into a data set for datacenter diagnostics;
• the report module further operable to send the log file to the analytic controller;
• and
• a path module residing at the analytic controller, the path module operable to:
• maintain topology information for the datacenter that indexes lists of edges and nodes traversed between combinations of source, physical IP addresses and destination, physical IP addresses, in a connection based approach to packet switching imposing a list across multiple connections with a common combination,
• provide a list of edges and nodes traversed by applying the source, physical IP address and the destination, physical IP address to the topology information.

A method for tracking datacenter utilization, the method comprising:

• collecting, by flow agents residing at end hosts in a datacenter, transmission data reporting amounts of data transmitted between source-destination pairs in the datacenter;
• accessing, by the flow agents, additional information local to the end hosts about a distributed job executed by the datacenter;
• combining, by the flow agents, the transmission data with the additional information local to the end hosts in files specific to the individual end hosts;
• provisioning, by the flow agents, the files to a common analytic controller.

A system for job-centric tracking of datacenter usage, comprising:

• multiple end hosts comprising hardware networked in a datacenter, the multiple end hosts providing distributed computing resources for a distributed, parallel processing job;
• multiple flow agents, the multiple flow agents residing at different end hosts among the multiple end hosts, an individual flow agent comprising:
• a traffic module operable to determine an amount of data sent from a common source to a common destination;
• and
• a report module operable to combine and correlate the amount of data sent with a source-destination pair in a report;
• and
• an analytic controller operable to receive and aggregate reports from the multiple end hosts into a diagnostic data set for the datacenter, the analytic controller comprising a map module operable to map at least one of a source, virtual Internet Protocol (IP) address and a destination, virtual IP address to at least one of a source, physical IP address and a destination, physical IP address.