# US 2015/0106357 A1 · Configuring web crawler to extract web page information > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2015/0106357 A1 - Document ID: US20150106357A1 - Status: Published application - Kind: A1 - Application number: 14/523,731 - Filed: 2014-10-24 - Published: 2015-04-16 - Earliest claimed: 2011-07-22 - Assignees: Alibaba Group Holding Limited - Applicants: Alibaba Group Holding Limited - Inventors: Yiming Sun; Qi Qiang; Boyang Cai; Xiaojun Jin; Zongyuan Wu - Web page: https://patentgrep.com/patent/US20150106357A1 ## CPC - G06F 16/951: Indexing; Web crawling techniques - G06F 16/9577: {Optimising the visualization of content, e.g. distillation of HTML documents} - G06F 16/353: into predefined classes - G06F 16/954: Navigation, e.g. using categorised browsing - G06F 16/9566: {URL specific, e.g. using aliases, detecting broken or misspelled links} - G06Q 10/10: Office automation; Time management - G06Q 30/02: Marketing; Price estimation or determination; Fundraising - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR ## Abstract Web crawling configuration includes: obtaining a webpage comprising a plurality of receiving a user selection of a node in the webpage; presenting a set of web crawling configuration options pertaining to a web crawling action to be performed with respect to the node, the set of web crawling configuration options depending at least in part on a type of an element included in the node and comprising: a first option to perform a first web crawling action in the event that the node include a first type of the element; and a second option to perform a second web crawling action in the event that the node includes a second type of the element; receiving a user input specifying the web crawling configuration option; and storing user specified web crawling configuration option, performing the web crawling action on the node according to the user input, or both. ## Claims ### Claim 1 (canceled) ### Claim 2 A system for configuring web crawling, comprising: one or more processors to: obtain a webpage comprising a plurality of nodes; present the webpage to a user; receive a user selection of a node in the webpage, the node comprising at least one element; in response to the user selection of the node, present a web crawling configuration option pertaining to a web crawling action to be performed with respect to the node, the web crawling configuration option depending at least in part on a type of an element included in the node; receive a user input specifying the web crawling configuration options pertaining to the web crawling action to be performed with respect to the node and a user indication to specify repeatedly the web crawling action to be performed on the plurality of nodes; specify repeatedly the web crawling action to be performed for the plurality of nodes; and store user specified web crawling configuration options, perform the web crawling action on the plurality of nodes according to the user input, or both; and one or more memories coupled to the one or more processors, to provide the one or more processors with instructions. ### Claim 3 (depends on 2) The system of claim 2, wherein the one or more processors are further to receive another user selection of a second node in the webpage, and receive a user input specifying an option pertaining to a web crawling action to be performed with respect to the second node. ### Claim 4 (depends on 2) The system of claim 2, wherein the one or more processors are further to process the obtained webpage such that each node is configurable for web crawling. ### Claim 5 (depends on 2) The system of claim 2, wherein the one or more processors are further to present a preview of the performed web crawling action according to the user input. ### Claim 6 (depends on 2) The system of claim 2, wherein the webpage is presented to the user via a graphical user interface (GUI). ### Claim 7 (depends on 2) The system of claim 2, wherein the web crawling action comprises text extraction. ### Claim 8 (depends on 7) The system of claim 7, wherein the web crawling configuration options for text extraction comprise configurations for one or more of: a filter, address extraction, date extraction, extracted text name, extracted text description, and format. ### Claim 9 (depends on 2) The system of claim 2, wherein the web crawling configuration option comprises settings associated with a node that is a link and the web crawling action comprises downloading the link. ### Claim 10 (depends on 9) The system of claim 9, wherein downloading the link comprises obtaining a subpage; and the one or more processors are further to present a web crawling configuration option pertaining to a web crawling action to be performed with respect to one or more nodes on the subpage. ### Claim 11 (depends on 10) The system of claim 10, the one or more processors are further to present a navigation tree of webpages, and associated web crawling configuration options with the webpage in the navigation tree of webpages. ### Claim 12 (depends on 2) The system of claim 2, wherein the web crawling configuration option pertaining to the web crawling action comprises one or more of the following link processing settings: extracting link text, extracting alt text, and extracting link target URL. ### Claim 13 (depends on 2) The system of claim 2, wherein the web crawling configuration option comprises a setting for downloading an image, capturing the image, or both. ### Claim 14 (depends on 2) The system of claim 2, wherein the web crawling configuration option comprises a setting for an event associated with the node, wherein the event includes: click, double click, mouse hover, rolling wheel operation, or mouse exit. ### Claim 15 (depends on 2) The system of claim 2, wherein the web crawling configuration option comprises a setting for a form element comprising a text box, wherein the setting comprises input text. ### Claim 16 (depends on 2) The system of claim 2, wherein the web crawling configuration option comprises a setting for a form element comprising: a text box, a button, a radio button, or a check box. ### Claim 17 (depends on 2) The system of claim 2, the one or more processors are further to store a locator of the node. ### Claim 18 (depends on 17) The system of claim 17, wherein the locator of the node selected by the user is based on XPath. ### Claim 19 (depends on 2) The system of claim 2, wherein performing the web crawling action on the node comprises downloading the webpage to be crawled. ### Claim 20 (depends on 2) The system of claim 2, wherein performing the web crawling action on the node comprises rendering Javascript code of at least a portion of the webpage. ### Claim 21 (depends on 2) The system of claim 2, further comprising storing results of the performed web crawling action. ### Claim 22 A method for configuring web crawling, comprising: obtaining, using one or more computer processors, a webpage comprising a plurality of nodes; presenting the webpage to a user; receiving a user selection of a node in the webpage, the node comprising at least one element; in response to the user selection of the node, presenting a web crawling configuration option pertaining to a web crawling action to be performed with respect to the node, the web crawling configuration option depending at least in part on a type of an element included in the node; receiving a user input specifying the web crawling configuration options pertaining to the web crawling action to be performed with respect to the node and a user indication to specify repeatedly the web crawling action to be performed on the plurality of nodes; specifying repeatedly the web crawling action to be performed for the plurality of nodes; and storing user specified web crawling configuration options, performing the web crawling action on the plurality of nodes according to the user input, or both. ### Claim 23 A computer program product for configuring web crawling, the computer program product being embodied in a tangible non-transitory computer readable storage medium and comprising computer instructions for: obtaining, using one or more computer processors, a webpage comprising a plurality of nodes; presenting the webpage to a user; receiving a user selection of a node in the webpage, the node comprising at least one element; in response to the user selection of the node, presenting a web crawling configuration option pertaining to a web crawling action to be performed with respect to the node, the web crawling configuration option depending at least in part on a type of an element included in the node; receiving a user input specifying the web crawling configuration options pertaining to the web crawling action to be performed with respect to the node and a user indication to specify repeatedly the web crawling action to be performed on the plurality of nodes; specifying repeatedly the web crawling action to be performed for the plurality of nodes; and storing user specified web crawling configuration options, performing the web crawling action on the plurality of nodes according to the user input, or both.