Scalable Analysis Platform For Semi-Structured Data

A method of operating a data analysis system, the method comprising:

• retrieving objects from a data source, wherein each of the retrieved objects includes (i) data and (ii) metadata describing the data;
• dynamically creating a cumulative schema by, for each object of the retrieved objects:
• (i) inferring a schema from the object based on the metadata of the object and inferred data types of elements of the data of the object,
• (ii) creating a unified schema, wherein the unified schema describes both (a) the object described by the inferred schema and (b) a cumulative set of objects described by the cumulative schema, and
• (iii) storing the unified schema as the cumulative schema;
• and
• storing the data of each of the retrieved objects in a storage service.

A non-transitory computer-readable medium storing processor-executable instructions, the instructions comprising:

• retrieving objects from a data source, wherein each of the retrieved objects includes (i) data and (ii) metadata describing the data;
• dynamically creating a cumulative schema by, for each object of the retrieved objects:
• (i) inferring a schema from the object based on the metadata of the object and inferred data types of elements of the data of the object,
• (ii) creating a unified schema, wherein the unified schema describes both (a) the object described by the inferred schema and (b) a cumulative set of objects described by the cumulative schema, and
• (iii) storing the unified schema as the cumulative schema;
• and
• storing the data of each of the retrieved objects in a storage service.