Cleaning and organizing schemaless semi-structured data for extract, transform, and load processing

A method, comprising:

• inferring, by a system, an event-specific schema for event data including a plurality of events,
• wherein the event-specific schema defines an organizational structure for the event data based on common attributes among the plurality of events, and
• wherein the plurality of events include actions or occurrences in a data ecosystem;
• generating, by the system, a first dataset to be ingested into an extract, transform, load (ETL) pipeline,
• wherein the first dataset includes a subset of the event data included in a second dataset that satisfies one or more registration parameters related to an ETL use case, and
• wherein the event data included in the second dataset is partitioned using the organizational structure defined by the event-specific schema;
• and
• providing, by the system, the first dataset to an ETL system configured to ingest the first dataset into the ETL pipeline and process the first dataset based on the ETL use case.

A non-transitory computer-readable medium storing a set of instructions, the set of instructions comprising:

• one or more instructions that, when executed by one or more processors of a system, cause the system to:
• infer an event-specific schema, for event data including a plurality of events, that defines an organizational structure for the event data based on common attributes among the plurality of events,
• wherein the plurality of events include actions or occurrences in a data ecosystem;
• generate a first dataset to be ingested into an extract, transform, load (ETL) pipeline,
• wherein the first dataset includes a subset of the event data included in a second dataset that satisfies one or more registration parameters related to an ETL use case, and
• wherein the event data included in the second dataset is partitioned using the organizational structure defined by the event-specific schema;
• provide the first dataset to an ETL system configured to ingest the first dataset into the ETL pipeline and process the first dataset based on the ETL use case;
• and
• generate a report that includes information related to the common attributes or information related to the event-specific schema.