Techniques for clustering structurally similar web pages

A computer-implemented method for grouping structurally similar pages, comprising:

• normalizing each Uniform Resource Locator (URL) associated with each Web page in a set of Web pages, resulting in normalized URLs,
• wherein the step of normalizing each URL includes, for each URL:
• dividing the URL into a plurality of tokens to produce a sequence of the plurality of tokens;
• assigning each token of said plurality of tokens to a level based, at least in part, on the position of the token in the sequence of the plurality of tokens;
• wherein each normalized URL uniquely identifies the tokens of said corresponding URL at each level of said corresponding URL;
• computing level-specific variations, wherein said variation for a corresponding level is computed based on the number of distinct tokens, at said corresponding level, in the URLs that correspond to said corresponding level relative to the total number of URLs associated with said set of Web pages;
• forming a plurality of groups of said Web pages based, at least in part, on said respective variations, wherein each group of said plurality of groups includes URLs that correspond to a particular corresponding level;
• and
• reducing the number of said plurality of groups of Web pages by aggregating one or more of said groups based on features of said Web pages in said one or more groups;
• wherein the method is performed by one or more computing devices.

A computer-implemented method for grouping structurally similar pages, comprising:

• determining a number of levels represented by each Uniform Resource Locator (URL) associated with each Web page in a set of Web pages;
• normalizing each URL associated with each Web page in said set of Web pages by representing each URL as a set of token values, wherein each set of token values contains a token value for each determined level of the corresponding URL and uniquely identifies the portion of the corresponding URL at each determined level of the corresponding URL;
• computing level-specific variations by, for each level of a plurality of levels, comparing the tokens that are assigned to the level only to other tokens that have been assigned to the same level to compute a level-specific variation for the level;
• wherein the level-specific variations indicate a measurement of URL affinity at the levels;
• wherein the variation for a corresponding level is computed based, at least in part, on a ratio of (a) the number of distinct token values, at the corresponding level, in the normalized URLs that correspond to the corresponding level and (b) the total number of URLs associated with said set of Web pages;
• and
• forming a first group of Web pages by grouping Web pages associated with said URLs at the level having minimum variation;
• wherein the method is performed by one or more computing devices.

A computer-readable storage medium storing instructions for grouping structurally similar pages, which instructions, when executed by one or more processors, cause the one of more processors to perform the steps of:

• normalizing each Uniform Resource Locator (URL) associated with each web page in a set of Web pages, wherein the step of normalizing each URL includes, for each URL:
• dividing the URL into a plurality of tokens to produce a sequence of the plurality of tokens;
• assigning each token of said plurality of tokens to a level based, at least in part, on the position of the token in the sequence of the plurality of tokens;
• computing level-specific variations by, for each level of a plurality of levels, comparing the tokens that are assigned to the level only to other tokens that have been assigned to the same level to compute a level-specific variation for the level;
• wherein the level-specific variations indicate a measurement of URL affinity at the levels;
• forming a plurality of groups of said Web pages based, at least in part, on said variations;
• wherein the variation for a corresponding level is computed based, at least in part, on the number of distinct values of the tokens at the corresponding level relative to the total number of tokens at the corresponding level.

A computer-readable storage medium storing instructions for grouping structurally similar pages, which instructions, when executed by one or more processors, cause the one of more processors to perform the steps of:

• normalizing each Uniform Resource Locator (URL) associated with each Web page in a set of Web pages, resulting in normalized URLs,
• wherein the step of normalizing each URL includes, for each URL:
• dividing the URL into a plurality of tokens to produce a sequence of the plurality of tokens;
• assigning each token of said plurality of tokens to a level based, at least in part, on the position of the token in the sequence of the plurality of tokens;
• wherein each normalized URL uniquely identifies the tokens of said corresponding URL at each level of said corresponding URL;
• computing level-specific variations, wherein said variation for a corresponding level is computed based on the number of distinct tokens, at said corresponding level, in the URLs that correspond to said corresponding level relative to the total number of URLs associated with said set of Web pages;
• forming a plurality of groups of said Web pages based, at least in part, on said respective variations, wherein each group of said plurality of groups includes URLs that correspond to a particular corresponding level;
• and
• reducing the number of said plurality of groups of Web pages by aggregating one or more of said groups based on features of said Web pages in said one or more groups.

A computer-readable storage medium storing instructions for grouping structurally similar pages, which instructions, when executed by one or more processors, cause the one of more processors to perform the steps of:

• determining a number of levels represented by each Uniform Resource Locator (URL) associated with each Web page in a set of Web pages;
• normalizing each URL associated with each Web page in said set of Web pages by representing each URL as a set of token values, wherein each set of token values contains a token value for each determined level of the corresponding URL and uniquely identifies the portion of the corresponding URL at each determined level of the corresponding URL;
• computing level-specific variations by, for each level of a plurality of levels, comparing the tokens that are assigned to the level only to other tokens that have been assigned to the same level to compute a level-specific variation for the level;
• wherein the level-specific variations indicate a measurement of URL affinity at the levels;
• wherein the variation for a corresponding level is computed based, at least in part, on a ratio of (a) the number of distinct token values, at the corresponding level, in the normalized URLs that correspond to the corresponding level and (b) the total number of URLs associated with said set of Web pages;
• and
• forming a first group of Web pages by grouping Web pages associated with said URLs at the level having minimum variation.