Dynamic data-ingestion pipeline

An apparatus, comprising:

• one or more processors;
• memory;
• and
• a program module, wherein the program module is stored in the memory and, during operation of the apparatus, is executed by the one or more processors to ingest data from a source in a set of sources, the program module including:
• instructions for accessing predefined configuration instructions;
• instructions for generating a dynamic data-ingestion pipeline compatible with a Hadoop file system based on the predefined configuration instructions,
• wherein the dynamic data-ingestion pipeline includes a modular arrangement of operators from a set of operators that includes:
• an extraction operator for extracting the data of interest from the source, a converter operator for transforming the data, and a quality-checker operator for checking the transformed data;
• instructions for receiving the data from the source;
• and
• instructions for processing the data using the dynamic data-ingestion pipeline by:
• using the converter operator to transform the received data into a common format compatible with the Hadoop file system;
• wherein the data is processed in real-time as the data is received, without storing the data in memory for the purpose of subsequent ingestion processing, thereby reducing storage requirements on the memory.

A system, comprising:

• a processing module comprising one or more processors and a non-transitory computer readable medium storing instructions that, when executed by the one or more processors, cause the system to:
• access predefined configuration instructions;
• generate a dynamic data-ingestion pipeline compatible with a Hadoop file system based on the predefined configuration instructions, wherein the dynamic data-ingestion pipeline includes a modular arrangement of operators from a set of operators that includes:
• an extraction operator for extracting the data of interest from the source, a converter operator for transforming the data, and a quality-checker operator for checking the transformed data;
• receive the data from a source in a set of sources;
• and
• process the data using the dynamic data-ingestion pipeline by:
• using the converter operator to transform the received data into a common format compatible with the Hadoop file system;
• wherein the data is processed in real-time as the data is received, without storing the data in memory for the purpose of subsequent ingestion processing, thereby reducing storage requirements of the system.