A method for automating a transmission of partitionable search results from a search engine, and reducing latency of delivering the partitionable search results, the method comprising:
- receiving a search request from a client, the search request comprising at least one natural language query for searching a database, wherein a search engine is configured to search the database based on the at least one natural language query and provide a plurality of partitions of a set of search results responsive to the at least one natural language query, wherein the set of search results contain all search results from the database that are responsive to the at least one natural language query, wherein the natural language query is input at the client by a user;
- partitioning the set of search results into the plurality of partitions of the set of search results after the search request comprising the at least one natural language query is received from the client and after submitting an initial request to the search engine after the search request comprising the at least one natural language query is received from the client, wherein the specific form of the database is abstracted such that the search engine is capable of searching partitioned and unpartitioned databases and enables a size to be determined of the set of search results, wherein each partition from the plurality of partitions of the set of search results contains a fraction of the set of search results responsive to the at least one natural language query;
- allocating a set of processing threads for a set of processing tasks, each task being:
- assigned to retrieve a subset of search results from the set of search results, and configured to execute in parallel with other processing tasks in the set of processing tasks, wherein each subset of search results is one of the partitions from the plurality of partitions of the set of search results;
- and
- executing the set of processing tasks, the step of executing comprising, for each processing task of the set of processing tasks:
- submitting at least one request to the search engine to retrieve the assigned subset of search results,
- receiving a response from the search engine, the response comprising the assigned subset of search results,
- creating a response message for the client based on the response from the search engine, the response message comprising the assigned subset of search results and metadata related to each search result, wherein the response message repackages the assigned subset of search results and metadata by excluding unused and irrelevant information, and
- sending the response message to the client, wherein the processing tasks execute in parallel and asynchronously with respect to each other, causing a server to receive assigned subsets of search results in different amounts of time and concurrently with respect to each other,
- wherein the response message is sent to the client as soon as it has been created such that when a first assigned subset of search results takes longer to retrieve from the search engine than a second assigned subset of search results, a response message containing the second assigned subset of search results can be created and sent to the client before the first assigned subset of search results is received at the server,
- wherein the set of processing tasks execute in parallel and continuously send a plurality of response messages to the client until all search results responsive to the at least one natural language query have been retrieved from the search engine and sent to the client resulting in the plurality of partitions of the set of search results responsive to the at least one natural language query being sent to the client,
- wherein the client begins parsing and displaying each response message of the plurality of response messages once they are received by the client and before the server has finished sending response messages,
- wherein all search results are received without loading an additional page or initiating a page refresh.