Direct cloud storage intake and upload architecture

A method of gathering and storing data in a cloud based architecture, comprising:

• gathering, on an enterprise side of a data collection system, a plurality of periodically generated events defined as documents in an unstructured database syntax;
• accumulating the gathered documents in an aggregation defined by part files having a columnar representation of the documents;
• uploading the aggregation to a cloud storage repository, the cloud storage repository configured for storing a plurality of the aggregations for responsiveness to a query server for satisfying query requests, the upload bypassing the query server while the events remain queryable from the query server;
• and
• generating the query requests remotely from the query server for accessing the cloud storage repository, the query requests generated based on header files corresponding to each of the part files, the header files including metadata and separate from the uploaded documents.

In a bigdata query processing environment having a customer premises computing system in network communication with a cloud storage repository for receiving queryable events from the customer premises computing system, the cloud storage repository responsive to query requests of the events from a query server, a non-transitory computer readable medium having computer program code encoded thereon, that, when executed by a processor, performs a method of uploading the queryable event data directly to the cloud storage repository, the method comprising:

• gathering, on an enterprise side of a customer premises computing system, a plurality of periodically generated events, each event defining a document, each document responsive to a query received at a query server side, the data collection system being in network communication with the query server;
• aggregating a portion of the plurality of events into a part file, each part file storing a subset of the gathered events arranged in a columnar format;
• generating header files corresponding to each of the part files;
• periodically, following arrangement of one or more part files to store aggregated events, uploading a sequence of the arranged part files to a cloud storage repository accessible by the query server, the cloud storage repository responsive to the query server for performing a query on the events contained in the uploaded part files, the upload bypassing the query server while leaving the parts files available for querying from the query server based on the header files.