Method and apparatus for representation of unstructured data

A computer-implemented method for representing and searching a document including unstructured data, the method comprising:

• generating, under control of the computer, a binary representation of the unstructured data in a plurality of documents;
• storing the binary representation of the unstructured data in a plurality of data structures, the data structures including:
• a first binary bit vector identifying a plurality of unstructured data stored in the plurality of documents;
• a plurality of second binary bit vectors, wherein for each of the plurality of unstructured data identified in the first binary bit vector, a corresponding second binary bit vector sets one or more bits for one or more position identifiers assigned to one or more instances of the associated unstructured data appearing in one or more of the plurality of documents, wherein the instance of an unstructured data appearing at the end of a first one of the plurality of documents is assigned a position identifier of n, and the instance of an unstructured data appearing at the beginning of a second one of the plurality of documents is assigned a position identifier of n+1, wherein n is an integer greater than 0;
• and
• a positional ID vector indicating a start position identifier of each word appearing at the beginning of each of the plurality of documents;
• receiving a search request including a search term;
• determining if a particular search term provided with the search request is identified in the first binary bit vector;
• if the particular search term is identified in the first binary bit vector, retrieving the corresponding second binary bit vector;
• identifying from the positional ID vector the start position identifier of the word at the beginning of a particular one of the plurality of documents to be searched;
• deducing from the positional ID vector an end position identifier of a word at the end of the particular one of the plurality of documents to be searched;
• identifying one or more bits set for one or more of the position identifiers in the retrieved secondary binary bit vector between the start position identifier and the end position identifier for identifying all instances of the search term occurring in the particular document;
• and
• generating, under control of the computer, an output based on the search.

A method for representing unstructured data included in a plurality of documents, the method comprising:

• parsing the plurality of documents;
• obtaining a unique word identifier for each unstructured data included in the plurality of documents;
• storing a first bit-value at each position of a first binary bit vector identified by each obtained unique word identifier;
• assigning a unique position identifier to each unstructured data included in the plurality of documents, wherein the unstructured data appearing at the end of a first one of the plurality of documents is assigned a position identifier of n, and the unstructured data appearing at the beginning of a second one of the plurality of documents is assigned a position identifier of n+1, wherein n is an integer greater than 0;
• retrieving a second binary bit vector for each of the unique word identifiers for which the first bit-value is set in the first binary bit vector;
• storing a second bit-value at each position of the retrieved second binary bit vector identified by one or more of the position identifiers assigned to one or more instances of the corresponding unstructured data identified by the unique word identifier;
• and
• setting a positional ID vector indicating a start position identifier of each word appearing at the beginning of each of the plurality of documents, and a word appearing at the beginning of a future document to the added to the plurality of documents.

A computer-implemented method for representing and searching a document including unstructured data, the method comprising:

• providing access to a plurality of document collections, each document collection storing a plurality of documents;
• representing the plurality of documents in a particular one of the document collections via a single text object, the single text object including a plurality of structures providing a binary representation of the plurality of the documents, the plurality of structures including:
• an inverted index including a word select vector and a word use set for indexing a plurality of words appearing in the plurality of documents, the word select vector setting a bit for each of the plurality of words, and the word use set including a position vector for each of the words having a bit set in the word select vector, the position vector setting a bit for each position identifier assigned to each instance of the corresponding word appearing in one or more of the plurality of documents, wherein the instance of a word appearing at the end of a first one of the plurality of documents is assigned a position identifier of n, and the instance of a word appearing at the beginning of a second one of the plurality of documents is assigned a position identifier of n+1, wherein n is an integer greater than 0;
• a positional ID vector indicating a start position identifier of each indexed word appearing at the beginning of each of the plurality of documents, and an indexed word appearing at the beginning of a future document to the added to the plurality of documents;
• and
• a document positions vector indicating, for each of the plurality of documents, an actual position in the corresponding document in which an instance of the indexed word occurs;
• receiving a search term;
• determining whether the bit corresponding to the search term is set in the word select vector;
• if the bit corresponding to the search term is set in the word select vector, retrieving the corresponding position vector from the word use set;
• selecting a first document to be searched from the plurality of documents in the particular document collection;
• retrieving the start position identifier of the indexed word at the beginning of the first document to be searched from the positional ID vector;
• deducing an end position identifier corresponding to an indexed word at the end of the first document to be searched from the positional ID vector;
• setting bits in a mask vector from a start position identified by the start position identifier to an end position identified by the end position identifier;
• performing a logical AND operation with the mask vector and the retrieved position vector for identifying all position identifiers for all instances of the search term occurring in the first document;
• and
• identifying actual locations of the identified instances of the search term in the first document from the document positions vector.