System and method for extracting and categorizing information from online sources

A method for avoiding redundant crawling by applying targeted URL filtering techniques, comprising:

• loading, by a processor, a home-page uniform resource locator (URL), wait until Document Object Model (DOM) events indicate that client-side scripts have finished rendering dynamic content, and capture rendered HTML of the home page;
• parsing, by the processor, the rendered HTML to collect one or more internal hyperlink reachable by a single click from the home page, filter out non-HTML resources, and produce a list of first-level internal URLs (depth-1 URLs), wherein the rendered HTML is parsed in a distributed real time data processing architecture comprising a master node;
• executing, by the processor, a classification model on the list of first-level internal URLs to predict a subset of URLs comprising information for generating tags, wherein predicting the subset of URLs comprises:
• extracting features from each of the first-level internal URL, including at least one of:
• URL structure, page title, and meta description;
• input the extracted features into the classification model, wherein the classification model is a fine-tuned Bidirectional Encoder Representations from Transformers (BERT) model that is trained on webpages from one or more company websites, including both structured and non-structured data, and wherein each webpage in training dataset is marked as one of relevant or not relevant;
• computing, via the classification model, a probability score indicating a likelihood if a first-level internal URL of the list of first-level internal URLs comprising information for generating the tags;
• predicting, based on the computed probability score, the subset of URLs with probability scores exceeding a predefined threshold for the tag generation;
• and
• continuously fine tuning the classification model with the predicted subset of URLs for next iterations for refinement in the classification model's performance to improve a prediction accuracy of the classification model over a period of time;
• parsing a URL, by the processor, of the subset of URLs to obtain parsed text, wherein the master node is configured to generate a plurality of worker nodes and distribute the subset of URLs amongst the plurality of worker nodes, wherein the master node is configured to determine whether the URL in the subset of URLs has already been parsed before a worker node initiates the parsing, thereby avoiding redundant parsing;
• executing, on the parsed text, a machine learning model trained for adaptive segmentation to identify one or more sections in the parsed text;
• segmenting, by the processor, the one or more sections into one or more contextual segments and storing the one or more contextual segments based on a hierarchal relationship between the one or more sections;
• and
• generating, by the processor, the tags through contextual analysis by applying a large-language model (LLM) to the one or more contextual segments.