Dynamic resource scaling for web crawling
AssigneeSalesforce, Inc.
InventorJose Lejin P J
Life of this patent
Estimating the term…
Abstract
A system may monitor behavior of first web crawling bots traversing a plurality of web pages to generate behavior information associated with the plurality of first web crawling bots. The system may store at least a portion of the behavior information in a behavior matrix. The system may receive plain language configuration instructions that describe one or more rules for allocating processing resources associated with serving the plurality of web pages. The system may detect a presence of a second web crawling bot accessing one or more first web pages of the plurality of web pages. The system may allocate the processing resources for the plurality of web pages in accordance with the one or more rules and based at least in part on detecting the presence of the second web crawling bot.
Claims20
A method for data processing at an application server, comprising:
- monitoring behavior of a plurality of first web crawling bots traversing a plurality of web pages to generate behavior information associated with the plurality of first web crawling bots;
- storing at least a portion of the behavior information in a behavior matrix;
- receiving plain language configuration instructions that describe one or more rules for allocating processing resources associated with serving the plurality of web pages;
- detecting a presence of a second web crawling bot accessing one or more first web pages of the plurality of web pages;
- and
- allocating the processing resources for the plurality of web pages in accordance with the one or more rules and based at least in part on detecting the presence of the second web crawling bot.
An application server for data processing, comprising:
- one or more memories storing processor-executable code;
- and
- one or more processors coupled with the one or more memories and individually or collectively operable to execute the code to cause the application server to:
- monitor behavior of a plurality of first web crawling bots traversing a plurality of web pages to generate behavior information associated with the plurality of first web crawling bots;
- store at least a portion of the behavior information in a behavior matrix;
- receive plain language configuration instructions that describe one or more rules for allocating processing resources associated with serving the plurality of web pages;
- detect a presence of a second web crawling bot accessing one or more first web pages of the plurality of web pages;
- and
- allocate the processing resources for the plurality of web pages in accordance with the one or more rules and based at least in part on detecting the presence of the second web crawling bot.
A non-transitory computer-readable medium storing code for data processing, the code comprising instructions executable by one or more processors to:
- monitor behavior of a plurality of first web crawling bots traversing a plurality of web pages to generate behavior information associated with the plurality of first web crawling bots;
- store at least a portion of the behavior information in a behavior matrix;
- receive plain language configuration instructions that describe one or more rules for allocating processing resources associated with serving the plurality of web pages;
- detect a presence of a second web crawling bot accessing one or more first web pages of the plurality of web pages;
- and
- allocate the processing resources for the plurality of web pages in accordance with the one or more rules and based at least in part on detecting the presence of the second web crawling bot.
Dependencies are inferred from the claim text.