Methods and systems of function-specific tracing

A method of function-specific tracing in a distributed application, the method comprising:

• generating a trace profile by observing the processing of a set of one or more inputs to a target program, the trace profile corresponding to the control flow between a plurality of functions invoked in the processing of said inputs;
• evaluating the applicability of one or more user-specified pruning criteria, the criteria selected from a group including a number of invocations, the total time spent in a particular function, the number of times a function is called, and a depth of the function call stack;
• pruning the trace profile according to the applicable evaluated pruning criteria;
• modifying the target program of the distributed application to provide tracing information from the functions identified in the trace profile;
• loading, into a compute controller in a distributed computing system, the modified target program;
• executing, by the compute controller, the modified target program;
• emitting metadata from the modified target program according to the trace profile;
• and
• collecting emitted metadata by a monitoring program.

A system comprising:

• a trace profiler to generate a trace profile identifying one or more functions of a target program, wherein the trace profile identifies one or more functions to trace by observing the execution of the target program in response to a first set of one or more inputs, and correlates an invoked function address with a function name, and outputs a list of functions corresponding to the control structures traversed within the target program while evaluating the first set of one or more inputs;
• a pruner to modify the trace profile by removing or correlating one or more function invocations from the trace profile according to one or more applicable user-specified pruning criteria, the criteria selected from a group including a number of invocations, the total time spent in a particular function, the number of times a function is called, and a depth of the function call stack;
• a compiler operable to take an input representing the target program and the trace profile and output an instrumented target program;
• a module to load, into a controller module in a first node of distributed computing system, the instrumented target program;
• a controller operable to execute the instrumented target program responsive to the reception of a received set of inputs and output program execution metadata received during the operation of the program;
• and
• a monitor for receiving the program execution metadata.

A system for profiling distributed applications, the system comprising:

• first, second, and third computing nodes, the first, second, and third nodes communicably coupled via a network;
• a first instrumented program executed in a context on the first node, and a second instrumented program executed in a context on the second node, wherein the instrumented programs have been modified to record metadata associated with one or more monitored function invocations, the monitored functions corresponding to a set of functions identified in a first trace profile for the first instrumented program and a second trace profile for the second instrumented program, the trace profiles associating internal function identifiers with function names;
• a monitoring program executed in a context on the third node;
• wherein the monitoring program associates function call metadata reported from the first and second instrumented program with the multi-node calling tree associated with the execution of a distributed application comprising the first and second instrumented programs.