Search engine design and computational cost analysis
AssigneeYahoo! Inc.
InventorsRicardo BAEZA-YATES, Aristides GIONIS, Flavio JUNQUEIRA, Vassilis PLACHOURAS, Luca TELLOLI
Life of this patent
Estimating the term…
Abstract
A computer implemented system for search engine facility architecting and design. The system estimates the costs of power and networking based on system parameters, such as average CPU utilization, connection time, and bytes transferred over the network. Regional distribution of facilities may be evaluated to take into account the various parameters and optimize the cost and speed of the systems being designed. The parameters used in analyzing and formulating an architecture are independent of a particular indexing or query processing technique.
Claims18
A computer program product, comprising a computer usable medium having a computer readable program code embodied therein, said computer readable program code adapted to be executed to implement a method for designing a search engine system, said method comprising:
- establishing a target latency for queries of a search processing system that services queries from a first geographic area and a second geographic area distant from the first geographic area;
- receiving a proposed topology for the search processing system;
- receiving a proposed location for a first site to service queries of the first and second geographic areas;
- receiving a proposed location for a second site to service queries of the first and second geographic areas, the first site being geographically distant from the second site;
- determining a power cost for power consumption of the first site by estimating power consumption of crawling operations of the first site;
- determining a power cost for power consumption of the first site by estimating power consumption of query processing operations of the first site;
- determining a power cost for power consumption of the second site by estimating power consumption of crawling operations of the second site;
- determining a power cost for power consumption of the second site by estimating power consumption of query processing operations of the second site;
- and
- calculating an overall operating cost of the search processing system from the power costs given the target latency, geographic areas to be served, proposed topology and locations.
A computer system configured to:
- receive a target query volume;
- calculate the cost of operation for a proposed distributed search system comprising at least one search repository site geographically distant from a second search repository site;
- calculate the cost of networking the search repository sites of the distributed search system;
- calculate the cost of operation for a proposed centralized search system;
- and
- determine whether the cost of operation of the proposed distributed system is greater or less than the cost of operation of the proposed centralized system.
A computer system configured to:
- calculate a cost of operation for a first proposed distributed search system comprising at least one search repository site geographically distant from a second search repository site of the first proposed system;
- calculate the cost of networking the search repository sites of the first distributed search system;
- calculate a cost of operation for a second proposed distributed search system comprising at least one search repository site geographically distant from a second search repository site of the second proposed system;
- calculate the cost of networking the search repository sites of the second distributed search system;
- and
- determine whether the cost of operation of the first proposed distributed system is greater or less than the cost of operation of the second proposed distributed system.
A computer program product, comprising a computer usable medium having a computer readable program code embodied therein, said computer readable program code adapted to be executed to implement a method for designing a search engine system, said method comprising:
- receiving an estimate for an overall query load for the search engine system or a portion thereof;
- and
- determining the cost of servicing the estimated query load by:
- (1) estimating a fraction of the overall query load that will be serviced by each of a plurality of geographically separated and distinct facilities;
- and
- (2) estimating the power consumption for the plurality of geographic locations.
A computer program product, comprising a computer usable medium having a computer readable program code embodied therein, said computer readable program code adapted to be executed to implement a method for designing a search engine system, said method comprising:
- determining a sum of power costs for at least two designs, each design having a different number of nodes from the other designs;
- determining a sum of bandwidth costs for the at least two designs, each design having a different number of nodes from the other designs;
- and
- determining an optimal number of nodes for the search engine system.
Dependencies are inferred from the claim text.