System and method transforming source data into output data in big data environments

A computer-based system, comprising:

• a processor;
• a tangible, non-transitory memory configured to communicate with the processor, the tangible, non-transitory memory having instructions stored thereon that, in response to execution by the processor, cause a transformation platform to perform operations comprising:
• receiving, by the transformation platform, a request to derive a requested output variable, wherein the request includes proposed logic to derive output data for the requested output variable from a source variable,
• wherein the transformation platform includes a processor and a tangible, non-transitory memory configured to communicate with the processor;
• comparing, by the transformation platform, the proposed logic to existing logic to determine the proposed logic is new, wherein the comparing comprises:
• generating, by the transformation platform, metadata that describes the proposed logic;
• looking up, by the transformation platform, metadata that describes the existing logic in a metadata store;
• and
• comparing, by the transformation platform, the metadata of the proposed logic to the metadata of the existing logic;
• generating, by the transformation platform, a transformation code in response to the proposed logic being new, wherein the transformation code is configured to execute the proposed logic;
• applying, by the transformation platform using the transformation code that is dynamically generated and includes machine executable code, the proposed logic to a source variable to generate the requested output variable;
• analyzing, by the transformation platform, existing scheduled tasks;
• scheduling, by the transformation platform and based on the existing scheduled tasks, the transformation code for execution at a predetermined time;
• executing, by the transformation platform, the transformation code to generate output data for the requested output variable at the predetermined time;
• returning, by the transformation platform and in response to the executing the transformation code, at least one of an existing output variable, a location of the existing output variable, or a copy of the existing output variable rather than a newly derived of the requested output variable based on the request;
• applying, by the transformation platform, the proposed logic from the request for the requested output variable to the identified source variable;
• and
• managing, by the transformation platform, the output data by evaluating logic to limit duplicative output data.

An article of manufacture including a non-transitory, tangible computer readable storage medium having instructions stored thereon that, in response to execution by a processor of a transformation platform, cause the transformation platform to perform operations comprising:

• receiving, by the transformation platform, a request to derive a requested output variable, wherein the request includes proposed logic to derive output data for the requested output variable from a source variable,
• wherein the transformation platform includes a processor and a tangible, non-transitory memory configured to communicate with the processor;
• comparing, by the transformation platform, the proposed logic to existing logic to determine the proposed logic is new, wherein the comparing comprises:
• generating, by the transformation platform, metadata that describes the proposed logic;
• looking up, by the transformation platform, metadata that describes the existing logic in a metadata store;
• and
• comparing, by the transformation platform, the metadata of the proposed logic to the metadata of the existing logic;
• generating, by the transformation platform, a transformation code in response to the proposed logic being new, wherein the transformation code is configured to execute the proposed logic;
• applying, by the transformation platform using the transformation code that is dynamically generated and includes machine executable code, the proposed logic to a source variable to generate the requested output variable;
• analyzing, by the transformation platform, existing scheduled tasks;
• scheduling, by the transformation platform and based on the existing scheduled tasks, the transformation code for execution at a predetermined time;
• executing, by the transformation platform, the transformation code to generate output data for the requested output variable at the predetermined time;
• returning, by the transformation platform and in response to the executing the transformation code, at least one of an existing output variable, a location of the existing output variable, or a copy of the existing output variable rather than a newly derived of the requested output variable based on the request;
• applying, by the transformation platform, the proposed logic from the request for the requested output variable to the identified source variable;
• and
• managing, by the transformation platform, the output data by evaluating logic to limit duplicative output data.