System and method for crawling

A method of (for) crawling, wherein the method includes using a computer system for executing data processing tasks, wherein the method comprises:

• (i) receiving at least one Uniform Resource Identifier;
• (ii) retrieving source information associated with the at least one Uniform Resource Identifier, wherein the source information includes a pool of data elements;
• (iii) determining at least one relevant data element from the pool of data elements, wherein determining the at least one relevant data element includes
• identifying at least one attribute associated with each data element in the pool of the data elements,
• analyzing the at least one identified attribute, based on predefined qualifier conditions, for detecting a relevance factor for the each data element, and
• using the relevance factor to determine the at least one relevant data element from the pool of data elements;
• (iv) analyzing the at least one relevant data element to determine an importance factor associated therewith;
• and
• (v) assigning a chronological score to each of the at least one relevant data element based on the determined importance factor thereof;
• and
• (vi) crawling each of the at least one relevant data element based on the assigned chronological score thereof.

A computer readable medium, containing program instructions for execution on a computer system, which when executed by a computer, cause the computer to perform method steps of a method of (for) crawling, the method comprising the steps of:

• receiving at least one Uniform Resource Identifier;
• retrieving source information associated with the at least one Uniform Resource Identifier, wherein the source information includes a pool of data elements;
• determining at least one relevant data element from the pool of data elements, wherein determining the at least one relevant data element includes:
• identifying at least one attribute associated with each data element in the pool of the data elements,
• analyzing the at least one identified attribute, based on predefined qualifier conditions, for detecting a relevance factor for the each data element, and
• using the relevance factor to determine the at least one relevant data element from the pool of data elements;
• analyzing the at least one relevant data element to determine an importance factor associated therewith;
• assigning a chronological score to each of the at least one relevant data element based on the determined importance factor thereof;
• and
• crawling each of the at least one relevant data element based on the assigned chronological score thereof.