Scalable storage and processing of hierarchical documents

A computer-readable storage medium having computer-executable instructions stored thereon, the instructions when executed by a processor causing the processor to implement a method for processing a data stream embodying a hierarchically structured document, the method comprising:

• querying the hierarchical structure of the document embodied in the data stream;
• determining an offset of the data stream, the offset determined during the querying, and the offset including one or more bits;
• partitioning said data stream into respective fixed length segments utilizing said queried hierarchical structure and the offset to determine a respective length of each fixed length segment;
• processing said fixed length segments in a pipeline fashion, the processing the fixed length segments including decoding the fixed length segments;
• parsing the decoded fixed length segments;
• partitioning the parsed fixed length segments into fragments, the fragments having at least one size, the at least one fragment size being cached in a memory of the computing device;
• inserting database persistence boundaries between the fragments;
• storing the fragments in a storage medium, the storage medium including a database and the at least one fragment size being determined in accordance with characteristics of the database, the characteristics including a native unit for holding data;
• creating a database table, the table comprising:
• meta data associated with the document;
• queries over the document and respective results;
• a first fragment of the document;
• sizes of all fragments of the document other than the first fragment;
• and
• storing the database table in the database.

A system for processing a data stream embodying a hierarchically structured document, said system comprising:

• at least one computing device;
• a receive pipeline for:
• receiving said data stream;
• querying the hierarchical structure of the document embodied in the data stream;
• determining an offset of the data stream, the offset determined during the querying, and the offset including one or more bits;
• partitioning said data stream into respective fixed length segments utilizing said queried hierarchical structure and the offset to determine a respective length of each fixed length segment;
• processing said fixed length segments in a pipeline fashion;
• decoding said fixed length segments;
• parsing said decoded fixed length segments;
• partitioning the parsed fixed length segments into fragments, the fragments having at least one size, the at least one fragment size being cached in a memory of the computing device;
• inserting database persistence boundaries between the fragments;
• storing the fragments in a storage medium, the storage medium including a database and the at least one fragment size being determined in accordance with characteristics of the database, the characteristics including a native unit for holding data;
• creating a database table, the table comprising:
• meta data associated with the document;
• queries over the document and respective results;
• a first fragment of the document;
• sizes of all fragments of the document other than the first fragment;
• and
• storing the database table in the database, said storage medium coupled to said receiving pipeline and coupled to a transmit pipeline;
• and
• said transmit pipeline for:
• receiving processed data from said storage medium;
• processing fixed length segments in a pipeline fashion.