System and method of crawling wide area computer network for retrieving contextual information

A method of crawling a wide area computer network for retrieving contextual information, wherein the method includes using a computer system, wherein the method comprises:

• receiving an input file, wherein the input file includes a plurality of Uniform Resource Identifiers, wherein each of the plurality of Uniform Resource Identifiers points to a corresponding web location in the wide area computer network;
• identifying, from the plurality of Uniform Resource Identifiers, accessible Uniform Resource Identifiers based on one or more parameters associated with each of the plurality of Uniform Resource Identifiers;
• obtaining one or more features associated with the accessible Uniform Resource Identifiers;
• executing an event for crawling the accessible Uniform Resource Identifiers to extract web content available in the corresponding web locations, wherein executing the event comprises:
• creating at least one resource cluster for executing the event, wherein the at least one resource cluster comprises one or more resources acquired based on the one or more features associated with the accessible Uniform Resource Identifiers;
• acquiring at least one electronic record file to be executed by the at least one resource cluster, wherein acquiring at least one electronic record file is based on the one or more features associated with the accessible Uniform Resource Identifiers;
• and
• crawling the accessible Uniform Resource Identifiers using the at least one resource cluster;
• processing the extracted web content to identify contextual data, wherein the contextual data is organized into one or more data structure;
• and
• aggregating the contextual data.

A computer readable medium, containing program instructions for execution on a computer system, which when executed by a computer, cause the computer to perform method steps for identifying at least one nascent topic related to a subject matter, the method comprising the steps of:

• receiving an input file, wherein the input file includes a plurality of Uniform Resource Identifiers, wherein each of the plurality of Uniform Resource Identifiers points to a corresponding web location in the wide area computer network;
• identifying, from the plurality of Uniform Resource Identifiers, accessible Uniform Resource Identifiers based on one or more parameters associated with each of the plurality of accessible Uniform Resource Identifiers;
• obtaining one or more features associated with the accessible Uniform Resource Identifiers;
• executing an event for crawling the accessible Uniform Resource Identifiers to extract web content available in the corresponding web locations, wherein executing the event comprises:
• creating at least one resource cluster for executing the event, wherein the at least one resource cluster comprises one or more resources acquired based on the one or more features associated with the accessible Uniform Resource Identifiers;
• acquiring at least one electronic record file to be executed by the at least one resource cluster, wherein acquiring at least one electronic record file is based on the one or more features associated with the accessible Uniform Resource Identifiers;
• and
• crawling the accessible Uniform Resource Identifiers using the at least one resource cluster;
• processing the extracted web content to identify contextual data, wherein the contextual data is organized into one or more data structure;
• and
• aggregating the contextual data.