A non-transitory computer-readable storage storing instructions which, when executed by one or more processors, cause:
- sending, by a search engine, to a first web server system that provides first web content, a first request for system configuration information that indicates hardware or software of the first web server system and that is separate from the first web content that is provided by the first web server system;
- after sending the first request, receiving, by the search engine, as a response to the first request, first system configuration information that is of a first type;
- wherein the first type is an information type in a particular group consisting of template information, session information, language-sub section information that indicates a first language of a first section of the first web content and a second language of a second section, of the first web content, that is different than the first section, identical content information, content interpretation information, microformat information that indicates a semantic meaning associated with (1) content on a first webpage hosted by the first web server system and (2) not content on a second webpage hosted by the first web server system, host information, error information, and application program information that indicates a first application program that is used for a third section of the first web content and a second application program that is used for a fourth section, of the first web content, that is different than the third section;
- updating, by the search engine, based on the first system configuration information that is stored separate from a first index, the first index with at least a subset of the first web content;
- in response to receiving, by the search engine, a first client request for content hosted on the first web server system, responding to the first client request by using the first index that has been updated based on the first system configuration information;
- sending, by the search engine, to a second web server system that provides second web content and that is different than the first web server system, a second request for system configuration information that pertains to hardware or software associated with the second web server system and that is separate from the second web content that is provided by the second web server system;
- after sending the second request, receiving, by the search engine, as a response to the second request, second system configuration information that is of a second type that is different than the first type and that is another type in the particular group;
- and
- updating, by the search engine, based on the second system configuration information, a second index with at least a subset of the second web content;
- in response to receiving, by the search engine, a second client request for content hosted on the first web server system, responding to the second client request by using the second index that has been updated based on the second system configuration information.