Method, system and device for identifying crawler data
AssigneeWangsu Science & Technology Co., Ltd.
InventorsZhiyong CHEN, Fengjie WANG, Zhiwen ZHAO
Life of this patent
Estimating the term…
Abstract
The present disclosure discloses a method, a system and a device for identifying crawler data. The method comprises: acquiring sitemap data of a target website and generating a vector graph of the sitemap data; acquiring session data of the target website, and mapping the session data into a subgraph in the vector graph based on requests contained in the session data; adding a session tag to the session data, where the session tag is configured to characterize whether the session data is crawler data; and training a preset classifier based on the session tag and the subgraph to obtain a trained classifier for distinguishing crawler data from non-crawler data.
Claims18
A method for identifying crawler data, comprising:
- acquiring sitemap data of a target website and generating a vector graph of the sitemap data;
- acquiring session data of the target website, and mapping the session data into a subgraph in the vector graph based on requests contained in the session data;
- adding a session tag to the session data, wherein the session tag is configured to characterize whether the session data is crawler data;
- and
- training a preset classifier based on the session tag and the subgraph to obtain a trained classifier for distinguishing crawler data from non-crawler data.
A device for identifying crawler data, comprising a memory and a processor, wherein the memory is configured to store a computer program, which, when executed by the processor, causes the processor to implement a method for identifying crawler data, the method comprising:
- acquiring sitemap data of a target website and generating a vector graph of the sitemap data;
- acquiring session data of the target website, and mapping the session data into a subgraph in the vector graph based on requests contained in the session data;
- adding a session tag to the session data, wherein the session tag is configured to characterize whether the session data is crawler data;
- and
- training a preset classifier based on the session tag and the subgraph to obtain a trained classifier for distinguishing crawler data from non-crawler data.
Dependencies are inferred from the claim text.