Method and system for performing topic-based aggregation of web content

A system for crawling webpages and aggregating topic-based web content, wherein the system comprises:

• a data repository configured to store a plurality of Uniform Resource Identifiers;
• a web crawling module including a communication interface for establishing a data connection with the data repository, wherein the web crawling module is configured to:
• receive the plurality of Uniform Resource Identifiers from the data repository, wherein each of the plurality of Uniform Resource Identifiers points to a web location related to a subject-matter;
• identify one or more portions of a webpage located at the web location relevant to the subject matter;
• crawl the one or more portions of the webpage to fetch web content, wherein the web content includes one or more elements;
• generate an electronic record file for storing:
• the web content including the one or more elements;
• and
• meta-information associated with the crawling of the webpage;
• identify a first type of element and at least a second type of element of the one or more elements associated with the web content stored in the electronic record file;
• process:
• the first type of element to obtain a topic-based web content, wherein the topic-based web content corresponds to a category associated with the subject-matter, and wherein the topic-based web content further comprises metadata associated with the topic-based web content, wherein the metadata includes a hash value or digest associated with the topic-based web content;
• and
• the at least one second type of element to obtain operational data, wherein the operational data comprises on or more hyperlinks retrieved from the web content stored in the electronic record file wherein the operational data comprises one or more hyperlinks retrieved from the web content stored in the electronic record file,
• and wherein the processing of the at least one second type of element by the web crawling module to obtain the operational data further comprises:
• assigning a score to each of one or more hyperlinks of the operational data wherein scoring of each of the hyperlinks includes calculating a probability of acquiring relevant web content form associated web location;
• and
• determining a crawling strategy and a hyperlink priority for crawling the one or more portions associated with each of the one or more hyperlinks based on the calculated probability to fetch the web content,
• and
• a database arrangement communicably coupled to the web crawling module, wherein the database arrangement is configured to aggregate the topic-based web content and the operational data, wherein the database arrangement stores the topic based web content based on the hash value associated with the topic based web content.

A non-transitory computer readable medium containing, program instructions for execution on a computer system, which when executed by a computer, cause the computer to perform method steps for method of crawling webpages and aggregating topic-based web content, the method comprising:

• a) receiving a plurality of Uniform Resource Identifiers, wherein each of the plurality of Uniform Resource Identifiers points to a web location related to a subject-matter;
• b) identifying one or more portions of a webpage located at the web location relevant to the subject matter;
• c) crawling the one or more portions of the webpage to fetch web content, wherein the web content includes one or more elements;
• d) generating an electronic record file for storing:
• the web content including the one or more elements;
• and meta-information associated with the crawling of the one or more portions of the webpage;
• e) identifying a first type of element and at least a second type of element of the one or more dements associated with the web content stored in the electronic record file;
• f) processing:
• the first type of dement to obtain a topic-based web content, wherein the topic-based web content corresponds to a category associated with the subject-matter, and wherein the topic-based web content further comprises metadata associated with the topic-based web content, wherein the metadata include a hash value or digest associated with the topic-based web content;
• and
• the at least one second type of element to obtain operational data wherein the operational data comprises one or more hyperlinks retrieved from the web content stored in the electronic record file;
• and wherein the processing of the at least one second type of dement to obtain the operational data further comprises:
• assigning a score to each of one or more hyperlinks of the operational data, wherein scoring of each of the hyperlinks includes calculating a probability of acquiring relevant web content form associated web location;
• and
• determining a crawling strategy and a hyperlink priority for crawling the one or more portions associated with each of the one or more hyperlinks based on the calculated probability to fetch the web content as per (d) to (f);
• g) aggregating the topic-based web content and the operational data into a database arrangement, wherein the database arrangement stores the topic based web content based on the hash value associated with the topic based web content.