Techniques for crawling dynamic web content

A method for crawling dynamic content, the method comprising:

• while crawling a first page:
• detecting a first form within the first web page;
• in response to detecting the first form, crawling the first form by constructing a plurality of requests for dynamic content from the server, based at least on the form;
• while crawling a second page that is different from the first page, subsequent to detecting the first form:
• detecting a second form within the second web page;
• determining that the second form is a duplicate version of the first form;
• wherein determining that the first form is a duplicate version of the second form comprises:
• determining that the first form comprises first metadata that is similar to second metadata that the second form comprises;
• in response to determining that the second form is a duplicate version of the first form, determining not to crawl the second form;
• wherein the method is performed by one or more processors at one or more computing devices.

A method of crawling content, the method comprising:

• identifying one or more JavaScript links within a page being crawled by a crawler, each JavaScript link being associated with executable JavaScript code;
• for each link of the one or more JavaScript links, determining whether or not the crawler should execute code associated with the link;
• wherein determining whether or not the web crawler should execute the code comprises:
• comparing information about the link to configuration information to determine the relevancy of the link, wherein the information about the link includes at least one of link text, anchor text, or information about an object near the link;
• based on the comparing, determining to execute particular code referenced in a particular JavaScript link;
• based on executing the particular code, identifying one or more additional pages for the crawler to crawl;
• wherein the method is performed by one or more processors at one or more computing device.