# US 10,482,083 B2 · Automated sequential site navigation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,482,083 B2
- Document ID: US10482083B2
- Status: Granted
- Kind: B2
- Application number: 15/287,668
- Filed: 2016-10-06
- Published: 2017-04-13
- Granted: 2019-11-19
- Earliest claimed: 2015-10-07
- Assignees: Capital One Services, LLC
- Applicants: Impossible Ventures, LLC
- Inventors: Robert Duke; Ryan Engle; Ryan Peck; Jonathan Coon
- Web page: https://patentgrep.com/patent/US10482083B2

## CPC

- G06Q 30/0629: {by pre-processing results, e.g. ranking or ordering results}
- G06F 16/245: Query processing
- G06F 16/95: Retrieval from the web
- G06F 16/986: {Document structures and storage, e.g. HTML extensions}
- H04L 67/01: Protocols
- H04L 67/02: based on web technology, e.g. hypertext transfer protocol [HTTP]
- H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46)
- H04L 67/564: Enhancement of application control based on intercepted application data
- H04L 67/567: Integrating service provisioning from a plurality of service providers
- H04L 67/60: Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources (admission control or resource allocation H04L47/70)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A website is navigated automatically in a sequential manner, for example by performing a series of specific network requests in a specified order, without necessarily using a browser, and without necessarily rendering each web page in the sequence. Interactions with a website are automatically recorded, stored, and analyzed. Requests and responses can be associated with particular input/output operations. Based on the analysis, the system can determine what data is expected by web servers, so that it can effectively simulate browser operations in connection with those web servers, and extract useful information from the data received in response to those operations. Thus, data extraction and/or other operations can be performed more efficiently and more quickly, while reducing the computing resources needed.

## Claims

### Claim 1

A computer-implemented method for automatically navigating a website, comprising:
at a processor on a client device, automatically recording a plurality of interactions and a plurality of network requests corresponding to one or more of the plurality of interactions that take place during performance of a web-based operation including communications with a web server, the plurality of interactions inputting data, the plurality of network requests including encoded data corresponding to the input data;
at a storage device, storing the recorded plurality of interactions and the recorded plurality of network requests;
at the processor, automatically analyzing the stored plurality of interactions and the stored plurality of network requests to determine an encoding scheme of the web server by:
determining a request, of the plurality of network requests, contains a first data of the input data of the plurality of interactions, and
swapping the first data in the request for a key to be used for future requests;
at the processor, automatically generating a rewritten network request based on the encoding scheme, the rewritten network request including substitute data in place of the encoded data by inserting the substitute data for the key in the request; and
at the processor, automatically executing the rewritten network request.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
automatically annotating the recorded plurality of network requests;
and wherein the storing the recorded plurality of network requests comprises storing the annotations.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein the automatically analyzing the stored plurality of interactions and the stored plurality of network requests comprises automatically analyzing the stored plurality of network requests based on the stored annotations.

### Claim 4 (depends on 2)

The computer-implemented method of claim 2, further comprising:
receiving user input during the recording of the plurality of network requests;
and wherein the automatically annotating the recorded plurality of network requests comprises automatically annotating the recorded plurality of network requests based on the received user input.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein the receiving the user input comprises receiving user input describing at least one selected from the group consisting of:
information displayed in response to the plurality of network requests;
information entered by a user in connection with the plurality of network requests; or
a description of an operation being performed in connection with the plurality of network requests.

### Claim 6 (depends on 4)

The computer-implemented method of claim 4, wherein the receiving the user input comprises receiving user input indicating that the plurality of network requests are associated with at least one selected from the group consisting of:
an add-to-cart request;
a display of pricing information; or
entry of a coupon code.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the automatically executing the rewritten network request comprises transmitting the rewritten network request to the web server.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, further comprising, prior to generating the rewritten network request, automatically identifying network requests of the plurality of network requests necessary to complete the web-based operation.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the web-based operation comprises displaying a plurality of web pages.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the web-based operation comprises a web-based purchase.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, wherein each network request comprises a communication between the client device and the web server.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, wherein the automatically executing the rewritten network request comprises automatically executing the rewritten network request without using a browser.

### Claim 13 (depends on 1)

The computer-implemented method of claim 1, wherein the automatically executing the rewritten network request comprises automatically executing the rewritten network request without rendering results for display.

### Claim 14 (depends on 1)

The computer-implemented method of claim 1, wherein the automatically executing the rewritten network request comprises automatically executing the rewritten network request without human interaction.

### Claim 15 (depends on 1)

The computer-implemented method of claim 1, further comprising automatically extracting information from the website while executing the rewritten network request.

### Claim 16

A non-transitory computer-readable medium for automatically navigating a website, comprising instructions stored thereon, that when executed by a processor on a client device, perform the steps of:
automatically recording a plurality of interactions and a plurality of network requests corresponding to one or more of the plurality of interactions that take place during performance of a web-based operation including communications with a web server, the plurality of interactions inputting data, the plurality of network requests including encoded data corresponding to the input data;
causing a storage device to store the recorded plurality of interactions and the recorded plurality of network requests;
automatically analyzing the stored plurality of interactions and the stored plurality of network requests to determine an encoding scheme of the web server by:
determining a request, of the plurality of network requests, contains a first data of the input data of the plurality of interactions, and
swapping the first data in the request for a key to be used for future requests;
automatically generating a rewritten network request based on the encoding scheme, the rewritten network request including substitute data in place of the encoded data by inserting the substitute data for the key in the request; and
automatically executing the rewritten network request.

### Claim 17 (depends on 16)

The non-transitory computer-readable medium of claim 16, further comprising instructions stored thereon, that when executed by the processor, perform the step of:
automatically annotating the recorded plurality of network requests;
and wherein the causing the storage device to store the recorded plurality of network requests comprises causing the storage device to store the annotations.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein the automatically analyzing the stored plurality of interactions and the stored plurality of network requests comprises automatically analyzing the stored plurality of network requests based on the stored annotations.

### Claim 19 (depends on 17)

The non-transitory computer-readable medium of claim 17, further comprising instructions stored thereon, that when executed by the processor, perform the step of:
causing an input device to receive user input during the recording of the plurality of network requests;
and wherein the automatically annotating the recorded plurality of network requests comprises automatically annotating the recorded plurality of network requests based on the received user input.

### Claim 20 (depends on 19)

The non-transitory computer-readable medium of claim 19, wherein the causing the input device to receive the user input comprises causing the input device to receive user input describing at least one selected from the group consisting of:
information displayed in response to the plurality of network requests;
information entered by a user in connection with the plurality of network requests; or
a description of an operation being performed in connection with the plurality of network requests.

### Claim 21 (depends on 19)

The non-transitory computer-readable medium of claim 19, wherein the causing the input device to receive the user input comprises causing the input device to receive user input indicating that the plurality of network requests are associated with at least one selected from the group consisting of:
an add-to-cart request;
a display of pricing information; or
entry of a coupon code.

### Claim 22 (depends on 16)

The non-transitory computer-readable medium of claim 16, wherein the automatically executing the rewritten network request comprises transmitting the rewritten network request to the web server.

### Claim 23 (depends on 16)

The non-transitory computer-readable medium of claim 16, further comprising instructions stored thereon, that when executed by the processor, perform the step of, prior to the generating rewritten network request, automatically identifying network requests of the plurality of network requests necessary to complete the web-based operation.

### Claim 24 (depends on 16)

The non-transitory computer-readable medium of claim 16, wherein the web-based operation comprises displaying a plurality of web pages.

### Claim 25 (depends on 16)

The non-transitory computer-readable medium of claim 16, wherein the web-based operation comprises a web-based purchase.

### Claim 26 (depends on 16)

The non-transitory computer-readable medium of claim 16, wherein each network request comprises a communication between the client device and the web server.

### Claim 27 (depends on 16)

The non-transitory computer-readable medium of claim 16, wherein the automatically executing the rewritten network request comprises automatically executing the rewritten network request without using a browser.

### Claim 28 (depends on 16)

The non-transitory computer-readable medium of claim 16, wherein the automatically executing the rewritten network request comprises automatically executing the rewritten network request without rendering results for display.

### Claim 29 (depends on 16)

The non-transitory computer-readable medium of claim 16, wherein the automatically executing the rewritten network request comprises automatically executing the rewritten network request without human interaction.

### Claim 30 (depends on 16)

The non-transitory computer-readable medium of claim 16, further comprising instructions stored thereon, that when executed by the processor, perform the step of automatically extracting information from the website while executing the rewritten network request.

### Claim 31

A system for automatically navigating a website, comprising:
a processor on a client device, configured to automatically record a plurality of interactions and a plurality of network requests corresponding to one or more of the plurality of interactions that take place during performance of a web-based operation including communications with a web server, the plurality of interactions inputting data, the plurality of network requests including encoded data corresponding to the input data; and
a storage device, communicatively coupled to the processor, configured to store the recorded plurality of interactions and the recorded plurality of network requests;
wherein the processor is further configured to perform the steps of:
automatically analyzing the stored plurality of interactions and the stored plurality of network requests to determine an encoding scheme of the web server by:
determining a request, of the plurality of network requests, contains a first data of the input data of the plurality of interactions, and
swapping the first data in the request for a key to be used for future requests;
automatically generating a rewritten network request based on the encoding scheme, the rewritten network request including substitute data in place of the encoded data by inserting the substitute data for the key in the request; and
automatically executing the rewritten network request.

### Claim 32 (depends on 31)

The system of claim 31, wherein the processor is further configured to perform the step of automatically annotating the recorded plurality of network requests;
and wherein the storing the recorded plurality of network requests comprises storing the annotations.

### Claim 33 (depends on 32)

The system of claim 32, wherein the automatically analyzing the stored plurality of interactions and the stored plurality of network requests comprises automatically analyzing the stored plurality of network requests based on the stored annotations.

### Claim 34 (depends on 32)

The system of claim 32, further comprising:
an input device, communicatively coupled to the processor, configured to receive user input during the recording of the plurality of network requests;
and wherein the automatically annotating the recorded plurality of network requests comprises automatically annotating the recorded plurality of network requests based on the received user input.

### Claim 35 (depends on 34)

The system of claim 34, wherein the receiving the user input comprises receiving user input describing at least one selected from the group consisting of:
information displayed in response to the plurality of network requests;
information entered by a user in connection with the plurality of network requests; or
a description of an operation being performed in connection with the plurality of network requests.

### Claim 36 (depends on 34)

The system of claim 34, wherein the receiving the user input comprises receiving user input indicating that the plurality of network requests are associated with at least one selected from the group consisting of:
an add-to-cart request;
a display of pricing information; or
entry of a coupon code.

### Claim 37 (depends on 31)

The system of claim 31, wherein the automatically executing the rewritten network request comprises transmitting the rewritten network request to the web server.

### Claim 38 (depends on 31)

The system of claim 31, wherein the processor is further configured to, prior to the generating rewritten network request, automatically identify network requests necessary to complete the web-based operation.

### Claim 39 (depends on 31)

The system of claim 31, wherein the web-based operation comprises displaying a plurality of web pages.

### Claim 40 (depends on 31)

The system of claim 31, wherein the web-based operation comprises a web-based purchase.

### Claim 41 (depends on 31)

The system of claim 31, wherein each network request comprises a communication between the client device and the web server.

### Claim 42 (depends on 31)

The system of claim 31, wherein the automatically executing the rewritten network request comprises automatically executing the rewritten network request without using a browser.

### Claim 43 (depends on 31)

The system of claim 31, wherein the automatically executing the rewritten network request comprises automatically executing the rewritten network request without rendering results for display.

### Claim 44 (depends on 31)

The system of claim 31, wherein the automatically executing the rewritten network request comprises automatically executing the rewritten network request without human interaction.

### Claim 45 (depends on 31)

The system of claim 31, further comprising automatically extracting information from the website while executing the rewritten network request.