Data edge platform for improved storage and analytics

A computer apparatus for storing a file in object storage, comprising:

• memory;
• and
• at least one processor coupled to the memory and configured to:
• receive, from an object storage system, a source file containing raw data;
• determine a plurality of symbols contained in the raw data of the source file;
• generate a symbol file associated with the source file, the symbol file comprising a sorted unique set of the plurality of symbols from the source file, wherein each symbol of the plurality of symbols is stored at a corresponding location within the symbol file;
• generate a locality file associated with the source file, the locality file comprising a plurality of location values referencing the symbol file, wherein a location value at a respective position within the locality file represents an occurrence in the source file of a corresponding symbol identified by a respective location value;
• store a converted file including the symbol file and the locality file associated with the source file in the object storage system, the converted file providing lossless compression of the source file;
• receive a search query regarding stored files in the object storage system;
• and
• display a result for the search query, the result for the search query returned based on querying at least one of the symbol file and the locality file of the converted file, the result of the search query reproducing the source file without accessing the source file.

A non-transitory computer-readable medium storing computer executable code for storing a file in object storage, comprising code to:

• receive, from an object storage system, a source file containing raw data;
• determine a plurality of symbols contained in the raw data of the source file;
• generate a symbol file associated with the source file, the symbol file comprising a sorted unique set of the plurality of symbols from the source file, wherein each symbol of the plurality of symbols is stored at a corresponding location within the symbol file;
• generate a locality file associated with the source file, the locality file comprising a plurality of location values referencing the symbol file, wherein a location value at a respective position within the locality file represents an occurrence in the source file of a corresponding symbol identified by a respective location value;
• store a converted file including the symbol file and the locality file associated with the source file in the object storage system, the converted file providing lossless compression of the source file;
• receive a search query regarding stored files in the object storage system;
• and
• display a result for the search query, the result for the search query returned based on querying at least one of the symbol file and the locality file of the converted file, the result of the search query reproducing the source file without accessing the source file.