Method for improving search engine efficiency

A system for improving the efficiency of a search engine in accessing, searching and retrieving information in the form of documents stored in document or content repositories, comprising:

• a processor;
• an indexing subsystem of the search engine that crawls the stored documents and generates an index, wherein applying a user search query to the index returns a result set of at least some query-matching documents, wherein the search engine comprises an array of search nodes hosted on one or more servers, wherein the array of search nodes comprises r rows and c columns characterized by classifying a query keyword in two dimensions, a first dimension being a posting list size and a second dimension an arrival rate that is determined using an arrival time of each query keyword, and wherein the index of the search engine is configured on a basis of one or more document properties, at least one of a fault-tolerance level, a required search performance, document meta-properties, and an optimal resource utilization;
• partitioning the index;
• replicating the index to create replicas that each comprise a same content as each of the other replicas;
• distributing the partitioned and replicated index over the array of search nodes such that index partitions and replicas thereof are assigned to said one or more servers hosting the array of search nodes, wherein distributing the index takes into account at least one of:
• posting lists size differences and posting lists popularity differences;
• and processing a search query on the basis of the distributed index.

A memory storing computer executable instructions that when executed perform actions, comprising:

• partitioning an index generated by a search engine that crawls stored documents;
• replicating the index to create replicas that each comprise a same content as each of the other replicas;
• distributing the partitioned and replicated index over an array of search nodes such that index partitions and replicas thereof are assigned to said one or more servers hosting the array of search nodes, wherein the array of search nodes comprises r rows and c columns characterized by classifying a query keyword in two dimensions, a first dimension being a posting list size and a second dimension an arrival rate that is determined using an arrival time of each query keyword, wherein distributing the index takes into account at least one of:
• posting lists size differences and posting lists popularity differences;
• and
• processing a search query on the basis of the distributed index.