Systems and methods for data scraping encrypted content sources comprising multi-layered encryptions and nested content
AssigneeDark Watch
InventorsNoel Thomas, James Snay
Life of this patent
Estimating the term…
Abstract
Systems and methods for uses and/or improvements to data scraping and/or data collection applications. As one example, systems and methods for web scraping applications that overcome multi-layered encryptions, content nesting, and/or other techniques for frustrating web scraping systems. For example, some websites, particularly those used in human trafficking, fraud, and/or other criminal enterprises, employ various encryption and/or encoding techniques to obstruct and frustrate web scraping systems, aiming to hide their data.
Claims20
A system for data scraping encrypted content sources comprising multi-layered encryptions and nested content using bifurcated data extraction routines that include initial analysis and tagging routines prior to dynamically selected extraction routines, the system comprising:
- one or more processors;
- and
- one or more non-transitory, computer-readable mediums comprising instructions that when executed by the one or more processors cause operations comprising:
- receiving a first user request to begin a secured session;
- authenticating the secured session;
- in response to authenticating the secured session, retrieving a first content source, wherein the first content source comprises a plurality of encrypted content masked by one or more of a plurality of data encryption types;
- executing a first routine, of a bifurcated data extraction routine, on the first content source by:
- determining a plurality of data elements in the first content source corresponding to a first data encryption type, of the plurality of data encryption types, for first encrypted content, wherein the plurality of data elements comprises one or more indicia of the first data encryption type;
- and
- determining a first encryption mapping of the first content source, wherein the first encryption mapping comprises a first location of the first data encryption type;
- in response to executing the first routine, executing a second routine, of the bifurcated data extraction routine, on the first content source to generate first non-encrypted content:
- selecting a first extraction function, from a plurality of extraction functions, based on the first data encryption type determined by the first routine;
- and
- applying the first extraction function to the first location;
- and
- generating for display, on a user interface, the first non-encrypted content during the secured session.
A method for data collection from encrypted content sources comprising multi-layered encryptions and nested content, the method comprising:
- retrieving a first content source, wherein the first content source comprises a plurality of encrypted content masked by one or more of a plurality of data encryption types;
- executing a first routine, of a bifurcated data extraction routine, on the first content source to determine a first data encryption type for first encrypted content in the first content source;
- in response to executing the first routine, executing a second routine, of the bifurcated data extraction routine, to select a first extraction function, from a plurality of extraction functions, based on the first data encryption type determined by the first routine;
- generating first non-encrypted content based on applying the first extraction function to the first encrypted content;
- and
- generating for display, on a user interface, the first non-encrypted content.
Dependencies are inferred from the claim text.