Data-driven detection of errors in data processing flows

A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a processor, cause the processor to:

• receive, by an application, runtime telemetry data comprising logs, exception events, and network packet n-tuple attributes from a plurality of components of a system that implements multi-stage processing workflows;
• map, by the application, the runtime telemetry data to a first processing template of a plurality of processing templates, the first processing template associated with the multi-stage processing workflow for a first processing operation performed by a subset of the plurality of components of the system;
• determine, by a model based on the mapped runtime telemetry data and the first processing template, an error associated with a first component of the subset of the plurality of components of the system, the model trained to:
• (i) determine error sources based on training logs, training exception events, and training network packet n-tuple attributes, and (ii) map errors to corrective actions;
• generate, by the model, a machine-executable corrective action mapped to the determined error;
• and
• initiate, by the application, performance of the machine-executable corrective action in the system.

An apparatus, comprising:

• a processor;
• and
• a memory storing instructions that, when executed by the processor, cause the processor to:
• receive, by an application, runtime telemetry data comprising logs, exception events, and network packet n-tuple attributes from a plurality of components of a system that implements multi-stage processing workflows;
• map, by the application, the runtime telemetry data to a first processing template of a plurality of processing templates, the first processing template associated with the multi-stage processing workflow for a first processing operation performed by a subset of the plurality of components of the system;
• determine, by a model based on the mapped runtime telemetry data and the first processing template, an error associated with a first component of the subset of the plurality of components of the system, the model trained to:
• (i) determine error sources based on training logs, training exception events, and training network packet n-tuple attributes, and (ii) map errors to corrective actions;
• generate, by the model based on the error, a machine-executable corrective action mapped to the determined error;
• and
• initiate, by the application, performance of the machine-executable corrective action in the system.