Method and system for building and contracting a linguistic dictionary

A system for building and contracting a linguistic dictionary, the linguistic dictionary comprising a list of surface forms and a list of normalized forms, each normalized form being associated with a surface form, the system comprising:

• a memory storing program information;
• and
• a central processing unit configured to execute the program information from the memory, wherein, upon execution of the program information, the central processing unit performs a method comprising the steps of:
• stemming the surface forms to derive a normalized form;
• applying a pruning algorithm to a trie-based data structure by populating the trie-based data structure with a surface form and generating a generic rule for applying to the surface form, wherein the generic rule describes how the surface form is to be processed, wherein the pruning step comprises traversing the trie-based data structure from node to node until the generic rule is found, comparing the generic rule with a most frequently occurring rule and if the generic rule is identical to the most frequently occurring rule removing the nodes associated with the generic rule;
• comparing each character of a surface form with each character of the surface form's normalized form, wherein the normalized form specifies a first form of a word and the surface form specifies a second and different form of the same word;
• responsive to the comparing, determining an edit operation for each character compared;
• and
• generating a transform code from the set of the edit operations specifying how to transform the surface form to its normalized form.

A computer program product for building and contracting a linguistic dictionary comprising:

• a computer readable storage medium having code stored thereon that, when executed by a system comprising a processor and a memory, causes the system to perform executable operations, the computer readable storage medium comprising:
• code for stemming the surface forms to derive a normalized form;
• code for applying a pruning algorithm to a trio-based data structure by populating the trio-based data structure with a surface form and generating a generic rule for applying to the surface form, wherein the generic rule describes how the surface form is to be processed, wherein the pruning step comprises traversing the trie-based data structure from node to node until the generic rule is found, comparing the generic rule with a most frequently occurring rule and if the generic rule is identical to the most frequently occurring rule removing the nodes associated with the generic rule;
• code for comparing each character of a surface form with each character of the surface form's normalized form, wherein the normalized form specifies a first form of a word and the surface form specifies a second and different form of the same word;
• code for, in response to the comparisons, determining an edit operation for each character compared;
• and
• code for generating a transform code from the set of the edit operations specifying how to transform the surface form to its normalized form.

A computer hardware system for building and contracting a linguistic dictionary, the linguistic dictionary comprising a list of surface forms and a list of normalized forms, each normalized form being associated with a surface form, comprising

• means for stemming the surface forms to derive a normalized form;
• means for applying a pruning algorithm to a trie-based data structure by populating the trie-based data structure with a surface form and generating a generic rule for applying to the surface form, wherein the generic rule describes how the surface form is to be processed, wherein the means for pruning comprises
• means for traversing the trie-based data structure from node to node until the generic rule is found,
• means for comparing the generic rule with a most frequently occurring rule and
• means, upon the generic rule is identical to the most frequently occurring rule, for removing the nodes associated with the generic rule;
• means for comparing each character of a surface form with each character of the surface form's normalized form, wherein the normalized form specifies a first form of a word and the surface form specifies a second and different form of the same word;
• means, in response to the comparing, for determining an edit operation for each character compared;
• means for generating a transform code from the set of the edit operations specifying how to transform the surface form to its normalized form;
• and
• means for storing the transform code within memory.