Data extraction approach for retail crawling engine

A system comprising:

• one or more processing devices;
• and
• one or more memory devices operably coupled to the one or more processing devices, the one or more memory devices storing executable code that, when executed by the one or more processing devices, causes the one or more processing devices to at least:
• retrieve a webpage including instructions for rendering a plurality of elements;
• render a modified version of the webpage in a virtual browser comprising processing each element of a first plurality of elements referenced in the webpage such that one or more resources referenced by an element are suppressed from being rendered in the modified version of the webpage;
• traverse the first plurality of elements of the webpage to identify selected elements from the plurality of elements based at least in part on a set of extraction instructions;
• apply the set of extraction instructions to the selected elements of the webpage to extract first selected data of the webpage according to the set of extraction instructions, wherein the selected data includes data describing a product;
• apply the set of extraction instructions to a second plurality of elements of a prior version of the webpage to extract second selected data of the webpage;
• generate, for a plurality of data types, metrics of the plurality of elements of the webpage at the inspection time, the metrics representing the at least a portion of the plurality of elements refrained from being rendered, the metrics based on characteristics of the plurality of elements and include at least:
• a first metric associated with a product description;
• a second metric associated with a product price;
• and
• a third metric associated with an image, wherein the third metric is generated at least in part by determining a vector representative of a composition of the image based at least in part on a segment mask associated with a feature represented in the image;
• determine, for each of the plurality of data types, differences between the metrics and prior metrics generated for the webpage;
• and
• in response to at least one of the differences exceeding a threshold:
• generate a side-by-side comparison including a comparison of at least one of a portion of the product description, the product price, or the image;
• and
• output the side-by-side comparison for display on a display device.

A system, comprising:

• one or more processors;
• and
• a memory coupled to the one or more processors and storing program instructions that, when executed by the one or more processors, cause the one or more processors to at least:
• retrieve, at a first time, a webpage including instructions for rendering a first plurality of elements;
• render a modified version of the webpage in a virtual browser comprising processing each element of a first plurality of elements referenced in the webpage such that one or more resources referenced by an element are suppressed from being rendered in the modified version of the webpage;
• traverse the first plurality of elements of the webpage to identify selected elements from the plurality of elements based at least in part on a set of extraction instructions;
• apply the set of extraction instructions to the selected elements of the webpage to extract first selected data of the webpage according to the set of extraction instructions, wherein the selected data includes data;
• generate, for a plurality of data types, first metrics of the first plurality of elements of the webpage at the first time, the first metrics representing the at least a portion of the first plurality of elements refrained from being rendered, wherein the first metrics are generated at least in part by determining a first vector representative of a first composition of a first image included in the first plurality of elements based at least in part on a first segment mask associated with a first feature represented in the first image;
• retrieve, at a second time, the webpage including the instructions for rendering a second plurality of elements;
• generate, for the plurality of data types, second metrics of the second plurality of elements of the webpage at the second time, the second metrics representing the at least a portion of the second plurality of elements refrained from being rendered, wherein the second metrics are generated at least in part by determining a second vector representative of a second composition of a second image included in the second plurality of elements based at least in part on a second segment mask associated with a second feature represented in the second image;
• determine, for at least one of the plurality of data types, a difference between the first metrics and the second metrics;
• and
• in response to the difference exceeding a threshold:
• generate a side-by-side comparison including a comparison of at least one of a portion of corresponding product descriptions, corresponding product prices, or corresponding images;
• and
• output the side-by-side comparison for display on a display device.