Deep pagination system

#The present specification generally relates to deep pagination in distributed databases according to various embodiments of the disclosure.

#A distributed database is a database in which storage devices are not all attached to a common processor. The distributed database may be stored in multiple computers, located in the same physical location, or may be dispersed over a network of interconnected computers. The multiple computers may be referred to as a cluster of nodes that may store a distributed index that is a collection of “documents” or other data objects distributed across the nodes. The distributed index may include a plurality of subsets called shards that are distributed over the various nodes and that store the various data objects. The distributed database may use a distributed search engine such as, for example, Elasticsearch™ to store, retrieve, and manage the various data objects stored in the distributed database. Distributed search engines often use pagination to retrieve the data objects when a search of the distributed database is performed.

#FIG. 1 is a block diagram illustrating a networked system according to an embodiment of the present disclosure;

#FIG. 2 is a block diagram illustrating a deep pagination system that is included in the networked system of FIG. 1 according to an embodiment of the present disclosure;

#FIG. 3A is flowchart illustrating a method of deep pagination according to an embodiment of the present disclosure;

#Embodiments of the present disclosure and their advantages are best understood by referring to the detailed description that follows. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures, wherein showings therein are for purposes of illustrating embodiments of the present disclosure and not for purposes of limiting the same.

#The present disclosure describes methods and systems for using distributed databases that require deep pagination to return requested data objects to a client system. As discussed above, distributed search engines may use pagination to provide a response to a request for data stored in a distributed database. Pagination provides many benefits when responding to the request for data. For example, pagination is used for easier handling of a response when there are many results. Pagination may make use of caching in the distributed database as the request to retrieve the next page will be served out of cache. With pagination, the amount of memory required on the server per request is lower as the server processes a portion of results instead of the whole result set and recovering from a failure during transmission is much easier if pagination is used as the server can resend only the last failed page.

#However, there are issues with pagination when the number of data objects that are to be returned for a request is relatively high. Distributed databases typically have a limit on the number of results returned per request. For example, the maximum number of data objects returned for a request may be 10,000 and these data objects may be paginated with a maximum of 500 results per page. However, this limit is not sufficient for various scenarios such as, for example, a merchant that wants to retrieve all of their transactions for a single day where that merchant may have over a million transactions. This maximum on the number of results is in place to prevent diminished performance on the distributed database system. If the number of results exceeds the limit, then deep pagination occurs, which is problematic. Deep pagination occurs when a request for data requires a number of pages into a dataset that begins to diminish the performance of a coordinating node that is coordinating the data results.