Method and apparatus for discovering and classifying polysemous word instances in web documents

An apparatus comprising:

• one or more processors;
• and
• a non-transitory computer readable storage medium storing one or more instructions which, when executed by the one or more processors, cause the one or more processors to perform:
• for each target word of target words, determining a set of overcorrelated pairs of said each target word and words in a plurality of documents, wherein an overcorrelated pair occurs when said each target word and a particular word in said plurality of documents have a first actual co-occurrence in portions of said plurality of documents that exceeds an expected probabilistic co-occurrence by at least a first threshold;
• wherein the first actual co-occurrence is determined by counting a number of blocks in said plurality of documents that contain both said each target word and the particular word;
• for said each target word of said target words and the set of the overcorrelated pairs determined for said each target word, determining overcorrelated triplets comprising said each target word and two words from said set of overcorrelated pairs, wherein an overcorrelated triplet occurs when said each target word and particular two words in said set of overcorrelated pairs have a second actual co-occurrence in said portions of said plurality of documents that exceeds said expected probabilistic co-occurrence by at least a second threshold;
• wherein said second actual co-occurrence is determined by counting a number of blocks in said plurality of documents that contain said each target word and the particular two words;
• wherein determining said overcorrelated triplet for said each target word and the particular two words comprises computing a ratio between said second actual co-occurence and a predicted proportion of blocks having said each target word and the particular two words;
• and
• storing data representing the overcorrelated triplets determined for said target words on one or more storage devices.

A non-transitory computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, cause the one or more processors to perform:

• for each target word of target words, determining a set of overcorrelated pairs of said each target word and words in a plurality of documents, wherein an overcorrelated pair occurs when said each target word and a particular word in said plurality of documents have a first actual co-occurrence in portions of said plurality of documents that exceeds an expected probabilistic co-occurrence by at least a first threshold;
• wherein the first actual co-occurrence is determined by counting a number of blocks in said plurality of documents that contain both said each target word and the particular word;
• for said each target word of said target words and the set of the overcorrelated pairs determined for said each target word, determining overcorrelated triplets comprising said each target word and two words from said set of overcorrelated pairs, wherein an overcorrelated triplet occurs when said each target word and particular two words in said set of overcorrelated pairs have a second actual co-occurrence in said portions of said plurality of documents that exceeds said expected probabilistic co-occurrence by at least a second threshold;
• wherein said second actual co-occurrence is determined by counting a number of blocks in said plurality of documents that contain said each target word and the particular two words;
• wherein determining an overcorrelated triplet for said each target word and the particular two words comprises computing a ratio between said second actual co-occurence and a predicted proportion of blocks having said each target word and the particular two words;
• and
• storing data representing the overcorrelated triplets determined for said target words on one or more storage devices.