# US 2012/0084077 A1 · Building and contracting a linguistic dictionary > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2012/0084077 A1 - Document ID: US20120084077A1 - Status: Published application - Kind: A1 - Application number: 13/316,511 - Filed: 2011-12-11 - Published: 2012-04-05 - Earliest claimed: 2005-06-29 - Assignees: International Business Machines Corporation - Inventors: HISHAM EMAD ELSHISHINY; EDEL GREEVY; PAI-FANG FRANNY HSIAO; ALEXEY NEVIDOMSKIY; ALEXANDER TROUSSOV; PAVEL VOLKOV - Web page: https://patentgrep.com/patent/US20120084077A1 ## CPC - G06F 40/242: Dictionaries - G06F 16/374: {Thesaurus} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A method for building and contracting a linguistic dictionary, the linguistic dictionary comprising a list of surface forms and a list of normalized forms, each normalized form being associated with a surface form, the method comprising the steps of: comparing each character of a surface form with each character of the surface form's normalized form; in response to the comparing step, determining an edit operation for each character compared; and generating a transform code from the set of the edit operations in order to transform the surface form to its normalized form. ## Claims ### Claim 1 A method for building and contracting a linguistic dictionary, the linguistic dictionary comprising a list of surface forms and a list of normalized forms, each normalized form being associated with a surface form, comprising: comparing, using a processor, each character of a surface form with each character of a normalized form of the surface form; in response to the comparing, determining an edit operation for each character compared; and generating a transform code from the set of the edit operations specifying how to transform the surface form to the normalized form, wherein the normalized form specifies a first form of a particular word, and the surface form specifies a second and different form of the particular word. ### Claim 2 (depends on 1) The method of claim 1, wherein the edit operation comprises deleting a character, retaining a character, inserting a character or replacing a character within the surface form for deriving the normalized form from the surface form. ### Claim 3 (depends on 1) The method of claim 1, further comprising determining a global frequency for each of the generated transform codes. ### Claim 4 (depends on 3) The method of claim 3, further comprising calculating whether a surface form comprises more than one transform code. ### Claim 5 (depends on 3) The method of claim 3 further comprising identifying, for each surface form having more than one transform code, whether at least one of the transform codes as having a higher global frequency than the other transform codes. ### Claim 6 (depends on 5) The method of claim 5, wherein the transform code with the higher global frequency replaces the transform code having a lower global frequency. ### Claim 7 (depends on 6) The method of claim 6, wherein the transform code having the higher global frequency is shared across surface forms. ### Claim 8 (depends on 1) The method of claim 1, wherein the linguistic dictionary comprises a trie-based data structure. ### Claim 9 (depends on 1) The method of claim 1, wherein the surface forms and normalized forms comprise Arabic surface forms and normalized forms. ### Claim 10 (depends on 7) The method of claim 7, further comprising calculating a usage rate of the highest occurring transform codes and selecting the transform code with the highest usage rate as the preferred transform code for storing. ### Claim 11 (depends on 1) The method of claim 1 further comprising the steps of stemming the surface forms to derive a normalized form and applying a pruning algorithm. ### Claim 12 (depends on 11) The method of claim 11 wherein the stemming step comprises generating a trie-based index for an information retrieval application. ### Claim at position 13 13-16. (canceled) ### Claim 17 A computer hardware system for building and contracting a linguistic dictionary, the linguistic dictionary comprising a list of surface forms and a list of normalized forms, each normalized form being associated with a surface form, comprising: a processor including a parsing component and an edit operation component, wherein the parsing component is configured to compare each character of a surface form with each character of a normalized form of the surface form; and the edit operation component is configured to determine an edit operation for each character compared, and; generate a transform code from the set of the edit operations specifying how to transform the surface form to the normalized form, the normalized form specifies a first form of a particular word, and the surface form specifies a second and different form of the particular word. ### Claim 18 (depends on 17) The system of claim 17, wherein the edit operation comprises deleting a character, retaining a character, inserting a character or replacing a character in the surface form for deriving the normalized form of the surface form. ### Claim 19 (depends on 17) The system of claim 17, wherein the processor further includes a determining component configured to determine a global frequency for each of the generated transform codes. ### Claim 20 (depends on 17) The system of claim 17, wherein the processor is configured to calculate whether a surface form comprises more than one transform code. ### Claim 21 (depends on 19) The system of claim 19, wherein the processor is configured to identify means, for each surface form having more than one transform code, whether at least one of the transform codes is determined as having a higher global frequency than the other transform codes. ### Claim 22 (depends on 21) The system of claim 21, wherein the transform code with the higher global frequency replaces the transform code having a lower global frequency. ### Claim 23 (depends on 22) The system of claim 22, wherein the transform code having the higher global frequency is shared across surface forms. ### Claim 24 (depends on 17) The system of claim 17, wherein the linguistic dictionary comprises a trie-based data structure. ### Claim 25 (depends on 17) The system of claim 17, wherein the surface forms and normalized forms comprise Arabic surface forms and normalized forms. ### Claim 26 (depends on 23) The system of claim 23, wherein the processor is configured to calculate a usage rate of the highest occurring transform codes and select the transform code with the highest usage rate as the preferred transform code for storing. ### Claim at position 24 27-31. (canceled) ### Claim 33 A computer program product comprising a computer usable storage medium having stored therein computer usable program code, which when executed on a computer hardware system, causes the computer hardware system to perform: comparing each character of a surface form with each character of a normalized form of the surface form; in response to the comparing, determining an edit operation for each character compared; and generating a transform code from the set of the edit operations specifying how to transform the surface form to the normalized form, wherein the normalized form specifies a first form of a particular word, and the surface form specifies a second and different form of the particular word.