Systems and methods for processing data for storing in a feature store and for use in machine learning

A computer-implemented method for processing data for use in machine learning models, the method comprising:

• receiving a request to generate a pipeline including two or more tasks, the request defining for each of the two or more tasks which features to ingest, which features to output, and instructions for processing the features;
• generating the pipeline including the two or more tasks based on the request;
• for one or more producer tasks of the two or more tasks:
• retrieving the ingestible features from one or more databases,
• processing the features based on the received instructions, and
• outputting curated features;
• for one or more consumer tasks of the two or more tasks:
• retrieving the ingestible features from a previous task in the pipeline upon which the consumer task depends,
• processing the features based on the received instructions, and
• outputting curated features;
• storing the curated features associated with one or more tasks of the two or more tasks in a feature store;
• and
• providing the stored curated features associated with the one or more tasks of the two or more tasks to a machine learning model for ingestion.

A computer-implemented system for processing data for use in machine learning models, the system comprising:

• a memory storing instructions;
• and
• one or more processors configured to execute operations comprising:
• receiving a request to generate a pipeline including two or more tasks and one or more sensors, the request defining for each of the two or more tasks which features to ingest, which features to output, and instructions for processing the features, and the sensors being configured to monitor at least one external trigger;
• generating the pipeline including the two or more tasks based on the request;
• for one or more producer tasks of the two or more tasks, at predetermined time intervals:
• retrieving the ingestible features from one or more databases,
• processing the features based on the received instructions, and
• outputting curated features;
• for one or more consumer tasks of the two or more tasks, at predetermined time intervals:
• retrieving the ingestible features from a previous task in the pipeline upon which the consumer task depends,
• processing the features based on the received instructions, and
• outputting curated features;
• storing the curated features associated with one or more tasks of the two or more tasks in a feature store;
• rendering a user interface based on the request, the user interface including a user-editable Directed Acyclic Graph (DAG) and a metadata browser specifying data about the stored curated features;
• and
• providing the stored curated features associated with the one or more tasks of the two or more tasks to a machine learning model for ingestion.