US 11,379,542 B1Granted

Advanced response processing in web data collection

AssigneeMetacluster Lt, Uab

InventorsMartynas Juravicius, Andrius Kuksta

Life of this patent

Estimating the term…

Loading patent…

Abstract

ADVANCED RESPONSE PROCESSING IN WEB DATA COLLECTION discloses processor-implemented apparatuses, methods, and systems of processing unstructured raw HTML responses collected in the context of a data collection service, the method comprising, in one embodiment, receiving raw unstructured HTML documents and extracting text data with associated meta information that may comprise style and formatting information. In some embodiments data field tags and values may be assigned to the text blocks extracted, classifying the data based on the processing of Machine Learning algorithms. Additionally, blocks of extracted data may be grouped and re-grouped together and presented as a single data point. In another embodiment the system may aggregate and present the text data with the associated meta information in a structured format. In certain embodiments the Machine Learning model may be a model trained on a pre-created training data set labeled manually or in an automatic fashion.

Claims20

Dependent claims

A method for classifying text blocks of a data collection response, comprising:

(a) receiving the data collection response that was scraped from a data collection target according to a data collection request wherein the request originates at a requesting user device;
(b) obtaining a plurality of text blocks from the data collection response:
for each text block in the plurality of text blocks;
(c) removing redundant text blocks from the plurality of text blocks resulting in a set of text blocks from the plurality of text blocks;
(d) obtaining a path describing a location of a respective text block from the set of text blocks within the data collection response;
(e) collecting, within the data collection response via the path, meta attributes describing the respective text block's display and functional characteristics within a page specified by the data collection response;
(f) deriving classification attributes from the respective text block, the path and the meta attributes;
(g) executing a trained machine learning classification model against the classification attributes to determine a classification for the respective text block;
(h) constructing a dataset with the set of text blocks and corresponding classifications determined in (g) for each text block in the set of text blocks;
and
(i) communicating the dataset to the requesting user device.

8 more elements

Dependencies are inferred from the claim text.