System and method of creating and using compact linguistic data

A compact linguistic data structure for a plurality of words, wherein the words are organized into groups, each group containing words that have a common prefix, the compact linguistic data structure comprising:

• an alphabet comprised of each unique character in the words;
• a character-mapping table for mapping each character in the alphabet to a character index;
• a substitution table for mapping sequences of characters from the words to substitution indexes;
• a plurality of word definition tables for storing the words, each word definition table storing each of the words included in one of the groups;
• and
• an offset table for locating the word definition tables, wherein for each of the common prefixes, the offset table contains a location of the word definition table which stores words starting with the common prefix,
• wherein each of the words in the word definition tables is encoded by replacing each character in the word with the character index to which the character is mapped by the character-mapping table, and by replacing each sequence of characters from the substitution table that appears in the word with the substitution index to which the sequence of characters is mapped by the substitution table;
• and
• wherein the common prefixes for words in each word definition table are removed.

A method of creating compact linguistic data, comprising steps of:

• creating a word-list comprising a plurality of words occurring most frequently in a corpus;
• sorting the words in the word-list alphabetically;
• creating a character-mapping table which maps characters in the words to character indexes;
• separating the words in the word-list into groups, wherein words in each group have a common prefix;
• creating a substitution table which maps character sequences in the words to substitution indexes;
• encoding the words into byte sequences by replacing characters in the words with the character indexes to which the characters are mapped by the character-mapping table, and by replacing character sequences in the words with the substitution indexes to which the character sequences are mapped by the substitution table;
• creating word definition tables and storing the encoded words in the word definition tables;
• creating an offset table for locating groups of encoded words;
• and
• storing the character-mapping table, the substitution table, the word definition tables, and the offset table.

The method of step 20, wherein:

• the step of creating the word-list comprises steps of:
• filtering the corpus into a plurality of filtered words;
• calculating absolute frequencies associated with the filtered words;
• and
• storing the filtered words occurring most frequently in the corpus and the absolute frequencies associated with the most frequently occurring filtered words in the word-list;
• and
• the method further comprises steps of:
• creating a plurality of relative frequencies by applying a normalization function to the absolute frequencies stored in the word-list;
• creating frequency sets associated with the groups of words;
• and
• storing the frequency sets in the word definition tables,
• wherein for each group, the associated frequency set contains a maximum relative frequency of words in the group, and contains, for each word in the group that is not associated with the maximum frequency, a percentage of the maximum relative frequency corresponding to the relative frequency associated with the word.