Method and system for encoding and accessing linguistic frequency data

A system for encoding linguistic frequency data, comprising:

• a processing unit for identifying a plurality of sets of character strings in a source text, each set comprising an n-gram including at least a first and a second character string, and, for each set, obtaining frequency data indicative of the frequency of the respective set in the source text, and
• an encoder that:
• creates a memory array f for containing the frequency data, n−1 pointer memory arrays p2 . . . pn for containing pointers, and n offset positional arrays r1–rn for containing indexing offsets,
• for each character string that is a first character string in at least one of the sets, assigns a memory position in memory array f to the respective character string and stores at said memory position the frequency data of each set comprising the respective character string as the first character string,
• groups the frequencies relating to n-grams that have the same first character string into a block within the array f,
• for each character string that is a second character string in at least one of the sets, assigns a memory position in pointer memory array p2 to the respective character string and stores at said memory position, for each set comprising the respective character string as second character string, a pointer pointing to a memory position in the memory array f assigned to the corresponding first character string of the respective set and having stored the frequency data of the respective set,
• groups the pointers relating to n-grams that have the same second character string together as a block within the pointer memory array p2,
• stores an offset position for each respective first character string in positional array r1 that indexes to the corresponding block in frequency array f relating to the first character string, and
• stores an offset position for each respective ith character string in positional array ri that indexes to the corresponding block in pointer array pi relating to the Ith character string.

Method of encoding linguistic frequency data, the method comprising:

• mapping each character string occurring in a source text to a numeric identifier identifying the character string,
• identifying a plurality of sets of n character strings in a source text, n being an integer number of at least 3, each set forming an n-gram comprising at least a first, a second and a third successive character string,
• for each set, obtaining frequency data indicative of the frequency of the respective set in the source text,
• creating a memory array f for containing the frequency data, and n−1 pointer memory arrays p2 . . . pn for containing pointers, and n offset positional arrays r1–rn for containing indexing offsets,
• for each character string that is a first character string in at least one of the sets, assigning a memory position in memory array f to the respective character string and storing at said memory position the frequency data of each set comprising the respective character string as the first character string,
• for each character string that is a second character string in at least one of the sets, assigning a memory position in pointer memory array p2 to the respective character string and storing at said memory position, for each set comprising the respective character string as the second character string, a pointer pointing to a memory position in the memory array f assigned to the corresponding first character string of the respective set and having stored the frequency data of the respective set, and
• for each character string that is an ith character string in at least one of the sets, for i=3 . . . n, assigning a memory position in an ith pointer memory array pi to the respective character string and storing at said memory position, for each set comprising the respective character string as the ith character string, a pointer pointing to a memory position in the memory array pi−1 assigned to the corresponding ith−1 character string of the respective set,
• wherein multiple pointers in the ith pointer memory array pi point to the same memory position within memory array pi−1 and only one chain of memory positions within f, p2 . . . pn uniquely define each n-gram.