US 7,599,931 B2 · Web forum crawler

Abstract

A crawling system crawls a web site initially in a pattern detection phase and subsequently in a pattern usage phase. The pattern detection phase attempts to identify patterns of references to pages that contain informational content of interest and patterns of references to pages that contain little informational content of interest. During the pattern usage phase, the crawling system crawls the web site. When the crawling system encounters a reference contained on an accessed page, the crawling system determines whether the reference matches a reference pattern. If the reference matches a reference pattern associated with pages that contain informational content of interest, the crawling system accesses the referenced page. If, however, the reference matches a reference pattern of pages with little informational content, then the crawling system discards that reference without accessing the referenced page.

Claims14

Dependent claims

A system with a processor and memory for crawling a site having pages, each page having a reference that identifies the page, each reference having tokens, comprising:

a grouping component that identifies groups of pages with similar content;
a pattern component that identifies a reference pattern of a group based on the references of the pages of the group, the reference pattern being identified by analyzing the tokens of the references of the pages of the group to identify sequences of tokens indicating a pattern of tokens within the references;
and
a decision component that, after encountering a reference that matches a reference pattern when crawling the site, decides whether to access the page of the encountered reference based on characteristics of the pages of the group of the matching reference pattern
wherein the components are implemented as computer-executable instructions stored in the memory for execution by the processor.

2 more elements

A system with a processor and memory for crawling a web site having web pages, each web page having a URL reference that identifies the web page, each URL reference having characters, comprising:

a group of informational web pages of the web site, the informational web pages being pages that are to be visited when crawling the web site;
a pattern component that identifies a URL reference pattern of the group of informational web pages of the web site based on the references of the web pages of the group, the URL reference pattern of a group representing a pattern of characters of URL references of the web pages of the group, the URL reference pattern including a wildcard character;
and
a crawling component that crawls the web site by
encountering URL references on web pages of the web site that reference other web pages of the web site;
after encountering a URL reference, determining whether the encountered URL reference matches the identified URL reference pattern based in part on the wildcard character of the URL reference pattern;
when the encountered URL reference matches the identified URL reference pattern, following the encountered URL reference to retrieve the web page identified by the encountered URL reference pattern;
and
when the encountered URL reference does not match the identified URL reference pattern, not following the encountered URL references
so that only encountered URL references that match the URL reference pattern of informational web pages are followed
wherein the components are implemented as computer-executable instructions stored in the memory for execution by the processor.

8 more elements

Dependencies are inferred from the claim text.

Web forum crawler

Life of this patent

Web forum crawler

Life of this patent

Abstract

Claims14