Scalable metadata index for a time-series database

#Many companies and other organizations operate computer networks that interconnect numerous computing systems to support their operations, such as with the computing systems being co-located (e.g., as part of a local network) or instead located in multiple distinct geographical locations (e.g., connected via one or more private or public intermediate networks). For example, distributed systems housing significant numbers of interconnected computing systems have become commonplace. Such distributed systems may provide back-end services or systems that interact with clients. For example, such distributed systems may provide database systems to clients. As the scale and scope of database systems have increased, the tasks of provisioning, administering, and managing system resources have become increasingly complicated. For example, the costs to search, analyze, and otherwise manage data sets can increase with the size and scale of the data sets.

#FIG. 1 illustrates an example system environment for a distributed time-series database including a scalable metadata index, according to some embodiments.

#FIG. 2 illustrates an example of hierarchical clustering of ingested time-series data in a distributed time-series database including a scalable metadata index, according to some embodiments.

#FIG. 3 illustrates an example of storage of time-series data using two-dimensional tiles in a hot tier in a distributed time-series database including a scalable metadata index, according to some embodiments.

#While embodiments are described herein by way of example for several embodiments and illustrative drawings, those skilled in the art will recognize that embodiments are not limited to the embodiments or drawings described. It should be understood, that the drawings and detailed description thereto are not intended to limit embodiments to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope as defined by the appended claims. The headings used herein are for organizational purposes only and are not meant to be used to limit the scope of the description or the claims. As used throughout this application, the word “may” is used in a permissive sense (i.e., meaning “having the potential to”), rather than the mandatory sense (i.e., meaning “must”). Similarly, the words “include,” “including,” and “includes” mean “including, but not limited to.”

#Embodiments of methods, systems, and computer-readable media for a scalable metadata index for a time-series database are described. A distributed time-series database hosted in the cloud may offer high availability and high throughput for both ingestion and queries of time-series data on behalf of multiple clients. The time-series database may include a set of ingestion routers that receive and spatially partition time-series data into a set of non-overlapping partitions. A particular time series may be defined by a schema that includes components such as one or more dimension names and a measure name. The time-series database may further include a set of storage nodes that process the partitioned time-series data. For example, the storage nodes may write elements of time-series data to one or more storage tiers, such as a “hot” tier that offers low-latency and high-availability write and read access to a set of distributed storage resources (e.g., as implemented using database clusters). In the hot tier, time-series data may be stored in two-dimensional tiles which are defined by spatial and temporal boundaries. The time-series database may further include a set of query processors that perform queries of the time-series data in the one or more storage tiers.

#To maintain high availability and high throughput for ingestion of time-series data, the time-series database may use a metadata index to identify the locations to which incoming time-series data points within particular time and space ranges are routed. Similarly, to maintain high availability and high throughput for queries of time-series data, the time-series database may use a metadata index to identify the locations to which queries for particular time and space ranges are routed. The metadata index may be maintained by a centralized service that authoritatively, consistently, and durably stores the locations and metadata of time-series data. The metadata index may be highly scalable and highly available. In some embodiments, the metadata index may support fast lookups (e.g., millisecond lookups) for billions of two-dimensional tiles and trillions of ingested data points per day. The metadata index may be built on a data store (e.g., a NoSQL data store) that provides scalability and durability, and the metadata index may distribute workload heat evenly across a caching fleet associated with the underlying data store.