Columnar storage and processing of unstructured data

In an unstructured data environment having an unstructured data file for query and retrieval by a query engine response to user queries, a method for transforming unstructured data in the unstructured data file to a columnar data file for query retrieval, comprising:

• identifying a data collection of unstructured data in the unstructured data file, the data collection having documents including named data fields of data values;
• transferring, from the data collection, the data values to a file storage medium based on the names of the data fields, the file storage medium having the data values denoted by a common data field name from the data collection;
• the data collection including a sequence of documents in the unstructured data file, each document in the sequence of documents defining the data values according to the data field names further comprising:
• generating, for each unique data field name in the data collection, a columnar data file, each columnar data file containing a sequence of values of similarly named data fields having a similar type and stored in a sequence of contiguous values according to their appearance in the documents of the data collection;
• generating a separate columnar data file based on a second named data field of the data field names in the data collection when at least one document in the data collection includes a different data field name, the generated columnar data file including similarly typed values of the named data field for each document in the data collection that includes a value for the named data field, the generated columnar data file stored based on a network link used to access the file and a retrieval speed of the network link;
• further comprising generating a separate columnar data file upon occurrence of:
• a new unique data field name occurs in a document of the data collection;
• a data field corresponding to a new compound field occurs in the document of the data collection, the new compound field having a plurality of subfields each having a name and a data value, further comprising defining a separate file for each named subfield in the compound field;
• a new data type occurs for a data field name already existing as a columnar data file for the collection;
• for each columnar data file generated, storing the data values at a position in the generated file which corresponds to a position of the document in the data collection from which the data values were transferred, position being preserved by null indicators corresponding to documents in the data collection devoid of a corresponding named field, each columnar data file storing the data values of the similarly named data fields at contiguous locations in the columnar data file, for query based retrieval of a plurality of data values independently of retrieval or parsing of the data values corresponding to other data field names in the data collection;
• generating for each columnar data file generated, a header indicative of the identified set for storing with the set in the columnar data file;
• performing a check for determining, for the generated columnar data file, whether the data type of the data values in the columnar data file from among numeric values and string values;
• when the data type is numeric values:
• storing, in the header, a file range denoting a minimum and maximum value of the numeric values in consecutive positions in the columnar data, the minimum and maximum values for determining a presence of a queried value in the columnar data file;
• when the data type is string values:
• computing a hash of each string value;
• sorting the computed hashes;
• and
• storing the computed hashes in the header for subsequent comparison with searched values;
• upon receipt of a request for querying the columnar data files:
• identifying a plurality of data field names for retrieval based on the query request;
• identifying a columnar data file storing values corresponding to one of the data field names of the plurality of data field names based on the query request;
• retrieving a first value from the identified columnar data file;
• and
• retrieving a second value from the identified columnar data file, the second value corresponding to the same data field name as the first value and retrieving the second value after the first value and before retrieving any other values corresponding to the plurality of data field names based on the query request, the first value and the second value stored in contiguous locations in the identified columnar data file.