Systems and methods for correlating errors to processing steps and data records to facilitate understanding of errors

A method for correlating errors in a data processing system, the method comprising:

• receive one or more commands from a computer program file;
• assign a first set of identifiers to different portions of the computer program file to link errors occurring during execution of one or more of a plurality of processing steps to corresponding portions of the computer program file;
• executing the plurality of processing steps to process data in a data processing system distributed over a plurality of nodes in a cluster, each of the processing steps having a unique tag, and each of the nodes being a computing device;
• composing a graph including vertices representing the plurality of processing steps;
• assigning tags to the vertices;
• access a policy that limits an amount of error information collected for each of the processing steps;
• based on a determination that a limit for a first processing step of the processing steps on the plurality of nodes has not been reached, collecting, from the plurality of nodes, in response to an error occurring on executing the first processing step, information about the error stored on the plurality of nodes, the information about the error including a first tag associated with one of the vertices representing the first processing step in the graph;
• processing the information about the error from the plurality of nodes to correlate the error to the first processing step based on the first tag that is associated with the first processing step and that is included in the information about the error stored on the plurality of nodes;
• generating, based on the processed information, correlation between the error and the first processing step, wherein the correlation between the error and the first processing step indicates a cause and a location of the error;
• present the generated correlation to a user.

A system for correlating errors in a data processing system, the system comprising:

• a processor and memory;
• and
• machine readable instructions, when executed by the processor and memory, configured to:
• receive one or more commands from a computer program file;
• assign a first set of identifiers to different portions of the computer program file to link errors occurring during execution of one or more of a plurality of processing steps to corresponding portions of the computer program file;
• execute the plurality of processing steps to process data in a data processing system distributed over a plurality of nodes in a cluster, each of the processing steps having a unique tag, and wherein each of the nodes is a computing device;
• compose a graph including vertices representing the plurality of processing steps;
• assign tags to the vertices;
• access a policy that limits an amount of error information collected for each of the processing steps;
• based on a determination that a limit for a first processing step of the processing steps on the plurality of nodes has not been reached, collect, from the plurality of nodes, in response to an error occurring on executing the first processing step, information about the error stored on the plurality of nodes, the information about the error including one or more records of data triggering the error and a first tag associated with one of the vertices representing the first processing step in the graph;
• process the information about the error from the plurality of nodes to correlate the error to the first processing step based on the first tag that is associated with the first processing step and that is included in the information about the error stored on the plurality of nodes;
• generate, based on the processed information, correlation between the error and at least one of the first processing step and the one or more records of data triggering the error, wherein the correlation between the error and at least one of the first processing step and the one or more records of data triggering the error indicates a cause and a location of the error;
• and
• present the generated correlation to a user.