Platform and source agnostic data processing for structured and unstructured data sources

A system comprising:

• a first data source comprising a data warehouse of structured data;
• a second data source comprising a data lake of unstructured data;
• a code library comprising:
• first code configured to perform at least one data processing task at a first data processing platform of a first type;
• and
• second code configured to perform the at least one data processing task at a second data processing platform of a second type that is different from the first type;
• one or more processors;
• and
• memory storing computer-readable instructions that, when executed by the one or more processors, cause the system to:
• receive a platform-agnostic data processing job comprising at least one platform-agnostic data processing task;
• receive a data source-agnostic data query comprising a data source alias;
• create, for a target data source, a data source-specific data query at least by replacing, using the alias mapping and based on the target data source, the data source alias with a data source identifier that indicates the target data source;
• obtain, for a target data processing platform and based on the at least one platform-agnostic data processing task, at least one platform-specific data processing task at least by retrieving, from the code library, platform-specific code configured to perform the at least one data processing task at the target data processing platform;
• create, for the target data processing platform and using the data source-specific data query and the at least one platform-specific data processing task, a restricted platform-specific data processing job that is restricted to execution within an interactive development environment (IDE) connected to the target data processing platform;
• obtain an unrestricted platform-specific data processing job configured for execution outside of the IDE at least by removing, from the restricted platform-specific data processing job, one or more configurations that restrict execution of the platform-specific data processing job to execution within the IDE, wherein the one or more configurations removed from the platform-specific data processing job comprise at least one of:
• at least one formatting element associated with the IDE;
• at least one input element associated with an input function of the IDE;
• or
• at least one output element associated with an output function of the IDE;
• and
• cause execution, outside of the IDE, of the unrestricted platform-specific data processing job at the target data processing platform, wherein the target data processing platform accesses, during execution of the unrestricted platform-specific data processing job and based on the data source-specific data query, the target data source.

A non-transitory computer-readable medium storing instructions that, when executed, cause:

• receiving a platform-agnostic data processing job comprising at least one platform-agnostic data processing task;
• receiving a data source-agnostic data query comprising a data source alias;
• creating, for a target data source, a data source-specific data query at least by replacing, using an alias mapping and based on the target data source, the data source alias with a data source identifier that indicates the target data source, wherein the alias mapping comprises a first mapping between a first data source alias and a first data source identifier of a first data source comprising a data warehouse of structured data, and a second mapping between a second alias and a second data source identifier of a second data source comprising a data lake of unstructured data;
• obtaining, for a target data processing platform and based on the at least one platform-agnostic data processing task, at least one platform-specific data processing task at least by retrieving, from a code library, platform-specific code configured to perform at least one data processing task at the target data processing platform, wherein the code library comprises first code configured to perform the at least one data processing task at a first data processing platform of a first type and second code configured to perform the at least one data processing task at a second data processing platform of a second type that is different from the first type;
• creating, for the target data processing platform and using the data source-specific data query and the at least one platform-specific data processing task, a restricted platform-specific data processing job that is restricted to execution within an interactive development environment (IDE) connected to the target data processing platform;
• obtaining an unrestricted platform-specific data processing job configured for execution outside of the IDE at least by removing, from the restricted platform-specific data processing job, one or more configurations that restrict execution of the restricted platform-specific data processing job to execution within the IDE;
• and
• causing execution, outside of the IDE, of the unrestricted platform-specific data processing job at the target data processing platform, wherein the target data processing platform accesses, during execution of the unrestricted platform-specific data processing job and based on the data source-specific data query, the target data source.