Back to results
US 11,082,321 B1Granted
Gossip-style database monitoring
AssigneeAmazon Technologies, Inc.
InventorMatthew Kyle Hulin
Life of this patent
Estimating the term…
Abstract
The present document describes systems and methods that monitor the health of a number of network-connected databases. In various embodiments, a health management subsystem measures one or more health parameters of a database and determines whether the database is operating properly. When the database is operating properly, a heartbeat associated with the database is generated. Heartbeat information is exchanged with other health management subsystems using a gossip-style protocol. Using the received and generated heartbeats, a health state table is provided that includes a health state for each database.
Claims20
A computer-implemented method comprising:
- measuring operational parameters that are associated with data replication functionality of a first database, the first database having a plurality of records that are selectively obtainable through queries to the first database, the first database storing data that is replicated from a second database;
- determining that the first database is properly replicating data from the second database to the first database based at least in part on the operational parameters, including by determining an amount of time taken to successfully replicate the data to the first database from the second database does not exceed a threshold time value;
- as a result of determining that the first database is properly replicating data from the second database to the first database, generating a heartbeat indicating proper replication operation performed by the first database;
- receiving, via a computer network, a set of heartbeats that include an incoming heartbeat for the first database and the second database, each heartbeat in the set of heartbeats indicating proper operation of replication functionality for an individual database;
- updating health-state table entries for the first and second databases in a health-state table, including by updating a last-update time to a time that the incoming heartbeat for the first database and the second database were respectively received;
- determining an operational-health state for the first database and the second database based at least in part on the generated heartbeat and the set of heartbeats, the operational-health state indicating an incommunicado state for a particular database when a heartbeat for the particular database has not been updated for a first threshold amount of time;
- generating a set of outgoing heartbeats for each database that the operational-health state does not indicate an incommunicado state;
- removing an entry associated with the particular database from the health-state table as a result of the particular database persisting in an incommunicado state for at least a second threshold amount of time;
- identifying a health management subsystem from a set of peer health management subsystems;
- and
- transmitting, via the computer network, the set of outgoing heartbeats to the health management subsystem, the set of outgoing heartbeats allowing the health management system to determine a replication status of the first database and the second database.
8 more elements
A system, comprising at least one computing device that implements one or more services, wherein the one or more services:
- determine that data is properly replicating from a master database to a slave database at a rate that does not exceed a threshold time value;
- update a heartbeat value for the slave database, the heartbeat value indicating that the data is being properly replicated from the master database to the slave database;
- receive a number of incoming heartbeat values, each incoming heartbeat value indicating a status of data replication for an individual slave database;
- update health-state table entries for the individual slave databases in a health-state table, including by updating a last-update time to a time that the incoming heartbeat for the individual slaved databases were respectively received;
- determine a health state for each slave database based at least in part on the updated heartbeat value and the incoming heartbeat values, the health state indicating a loss of a communication state for each database for which a heartbeat has not been obtained for a first threshold amount of time;
- delete an entry associated with a particular database from the health-state table as a result of the particular database indicating a loss of communication for at least a second threshold amount of time;
- generate a collection of outgoing heartbeats corresponding to those slave databases having a determined health state not indicating the loss of the communication state;
- and
- provide the collection of outgoing heartbeats to another service.
6 more elements
A non-transitory computer-readable storage medium having stored thereon executable instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to at least:
- determine that a database is receiving replicated data from a master database, the database being a read replica of the master database;
- as a result of determining that the database is replicating data at a rate that does not exceed a threshold time value, update a heartbeat value in a health-state table, the heartbeat value generated in response to determining that the database is replicating the data from the master database;
- update health-state table entries for individual databases in the health-state table, including by updating a last-update time to a time that an incoming heartbeat for the individual databases are respectively received;
- receive an incoming gossip message that includes a heartbeat value and a heartbeat identifier for the individual databases;
- update the health-state table based at least in part on the gossip message and the incoming gossip message, the health-state table indicating loss of communication with a database for which a heartbeat has not been obtained within a first threshold amount of time;
- delete an entry associated with a particular database from the health-state table as a result of the particular database indicating a loss of communication for at least a second threshold amount of time;
- generate, based at least in part on information stored in the health-state table, an outgoing gossip message that includes a heartbeat for each database that is properly replicating data from the master database;
- and
- send the outgoing gossip message over a computer network.
6 more elements
Dependencies are inferred from the claim text.