System and method for artificial intelligence base prediction of delays in pipeline processing

A method for predicting processing times of a pipeline, the method comprising:

• receiving a trigger object initiating the pipeline;
• initiating a machine learning (ML) system using an ML model associating features of the trigger with processing times at a plurality of stages of the pipeline;
• extracting features of the trigger object;
• training each of a plurality of ML subsystems of the ML system to provide a respective individual inference model;
• identifying and analyzing, using the respective individual inference model, feature patterns between delay and each of the one or more features;
• and
• inferring each processing time at each of the plurality of stages of the pipeline by processing the features using the ML system and based on the feature patterns, wherein processing the features comprises:
• computing, for each of the plurality of ML subsystems, a respective error rate;
• and
• deriving and storing the ML model by aggregating the individual inference models based on a particular type of the pipeline and weighting the corresponding subsystems according to their error rates.

A training system comprising:

• a processor;
• and
• a memory in communication with the processor and comprising instructions which, when executed by the processor, program the processor to:
• receive labelled data pertaining to a pipeline, the labelled data comprising trigger objects initiating the pipeline and target processing times corresponding to a plurality of stages of the pipeline;
• identify one or more features associated with the trigger objects;
• split the labelled data into a full training dataset and a testing dataset;
• in an ensemble machine learning (ML) system, train each of a plurality of ML subsystems using a respective partial dataset of the full training dataset, to provide a respective individual inference model;
• identify and analyze, using the respective individual inference model, feature patterns between delay and each of the one or more features;
• predict, using the respective individual inference model and based on the feature patterns, a respective processing time at each of the plurality of stages of the pipeline in terms of at least one feature;
• compute, for each of the plurality of ML subsystems, a respective error rate using the testing dataset;
• and
• derive and store an ML model by aggregating the individual inference models based on a particular type of the pipeline and weighting the corresponding subsystems according to their error rates.

A training system comprising:

• a processor;
• and
• a memory in communication with the processor and comprising instructions which, when executed by the processor, program the processor to:
• receive a trigger object initiating a pipeline;
• initiate a machine learning (ML) system using an ML model associating features of the trigger with processing times at a plurality of stages of the pipeline;
• extract features of the trigger object;
• train each of a plurality of ML subsystems of the ML system to provide a respective individual inference model;
• identify and analyze, using the respective individual inference model, feature patterns between delay and each of the one or more features;
• and
• infer each processing time at each of the plurality of stages of the pipeline by processing the features using the ML system and based on the feature patterns,
• wherein to process the features, the instructions further program the processor to:
• compute, for each of the plurality of ML subsystems, a respective error derive and store the ML model by aggregating the individual inference rate;
• and
• models based on a particular type of the pipeline and weighting the corresponding subsystems according to their error rates.