Systems and methods for common data ingestion

A method for common data ingestion comprising:

• in an information processing apparatus including at least a memory, a communication interface, and at least one computer processor:
• registering a project comprising a plurality of project data files, each project data file having a project data characteristic and a source file name that adheres to a predefined pattern;
• registering a plurality of data zones in a data reservoir for receiving project data files associated with the project, each data zone associated with a different data characteristic, wherein the data characteristic includes a data format, a retention setting, and a compression state, and wherein registering the plurality of data zones comprises generating, for each zone, a logical to physical data map for a physical data platform;
• receiving the project data files associated with the project into a common drop zone;
• instantiating a plurality of file-watcher threads at a data ingestion service of the processing apparatus, wherein each of the file-watcher threads is configured to monitor for a corresponding pattern;
• monitoring, by one file watcher thread of the plurality of file-watcher threads, for incoming data from the common drop zone that is associated with the predefined pattern;
• determining, by the one file-watcher thread of the plurality of file watcher threads, that the project data files are associated with the predefined pattern;
• triggering a predefined data processing job of the project data files based on the determining that the project data files are associated with the predefined pattern, wherein the predefined data processing job includes:
• identifying a personal information classification for each of the project data files, wherein the personal information classification is one of a first-level confidential field sensitivity level or a second-level field sensitivity level;
• performing an approval process based on the personal information classification;
• responsive to an approval based on the approval process, establishing a security policy for each project data file;
• routing each project data file from the common drop zone to one of the data zones based on the project data characteristic and the data characteristic for each data zone;
• and
• exporting the project data to at least one target.