Parameterized machine learning pipeline implemented using a lambda architecture

A system for developing and deploying a machine learning pipeline, the system comprising:

• a receiving module, that when executed by at least one processor, is configured to collect different types of data from one or more data sources, wherein the different types of data include batch data and real-time data;
• a reference module, that when executed by the at least one processor, is configured to continuously process the different types of data to build a modular machine learning pipeline including a plurality of module layers of a Lambda application, wherein each layer is configured to be deployed independently of the other layers and the reference module is built by combining the plurality of module layers, independently, wherein the plurality of layers comprises a batch layer, a real-time layer and an inference layer;
• a CI/CD pipeline module, that when executed by the at least one processor, is configured to deploy a source code into the modular machine learning pipeline, independently;
• wherein the reference module, that when executed by the at least one processor, is configured to be deployed as a single deployment machine learning pipeline for different use cases by revising the source code of a CI/CD pipeline, while a framework of the single deployment machine learning pipeline remains unchanged;
• wherein the plurality of layers are different from each other, and one of the plurality of layers is configured as a prediction-generating layer to generate, based on the different types of data without modifying the prediction data, prediction data that are imported, unidirectionally and independently, to at least two different layers of the plurality of layers, wherein the at least two different layers of the plurality of layers include the batch inference layer and the real-time inference layer configured to operate asynchronously with respect to the prediction-generating layer;
• and
• wherein the CI/CD pipeline is configured to build and test the source code before deploying the source code directly into at least one of the plurality of layers based on the source code, the CI/CD pipeline is configured to be integrated independently into each of the respective plurality of layers, and the CI/CD pipeline is configured to continuously and automatically introduce the source code into any stage of the respective plurality of layers that the CI/CD pipeline determines is an appropriate layer to receive the source code based on new changes to the source code;
• and
• wherein the CI/CD pipeline is configured to identify an error condition in the modular machine learning pipeline, ensure a continuous flow of new features by introducing the source code into any stage of the respective plurality of layers to correct the error condition in the modular machine learning pipeline.

A tangible computer-readable medium having stored thereon, computer executable instructions that, if executed by a computing device, cause the computing device to perform a method for developing and deploying a machine learning pipeline, the method comprising:

• collect different types of data, by at least one processor, from one or more data sources, wherein the different types of data include batch data and real-time data;
• continuously process, by at least one processor, the different types of data to build a plurality of layers of a Lambda application, wherein each layer is configured to be deployed independently of the other layers;
• build, by the at least one processor, a modular machine learning pipeline in a modifiable reference file, wherein the modular machine learning pipeline defines a framework comprising a no-code/low-code parameterized machine learning framework;
• build and integrate, by the at least one processor, a CI/CD pipeline to deploy a source code into the modular machine learning pipeline;
• deploy, by the at least one processor, the modular machine learning pipeline as a single deployment machine learning pipeline for different use cases by revising the source code of the CI/CD pipeline, while the framework of the single deployment machine learning pipeline remains unchanged;
• wherein the plurality of layers are different from each other, and one of the plurality of layers is configured as a prediction-generating layer to generate, based on the different types of data, prediction data that are imported, unidirectionally and independently without modifying the prediction data, to at least two different layers of the plurality of layers, and wherein the at least two different layers of the plurality of layers include a batch inference layer and a real-time inference layer configured to operate asynchronously with respect to the prediction-generating layer;
• and
• wherein the CI/CD pipeline is configured to build and test the source code before deploying the source code directly into at least one of the plurality of layers based on the source code, the CI/CD pipeline is configured to be integrated independently into each of the respective plurality of layers, and the CI/CD pipeline is configured to continuously and automatically introduce the source code into any stage of the respective plurality of layers that the CI/CD pipeline determines is an appropriate layer to receive the source code based on new changes to the source code;
• and
• wherein the CI/CD pipeline is configured to identify an error condition in the modular machine learning pipeline, ensure a continuous flow of new features by introducing the source code into any stage of the respective plurality of layers to correct the error condition in the modular machine learning pipeline.