Workflow for database system recovery using local and backed up resources

A method implemented in a data processing system for provisioning and operating databases, the method comprising:

• operating a database including a plurality of replicas associated with a tenant of the data processing system, the database comprises software for collecting, organizing, storing, accessing, and manipulating electronic data, the database including data structures for organizing and storing the electronic data, wherein a replica of the database represents a copy of the database, and wherein operating the database includes:
• operating a primary replica capable of processing both read and write operations;
• operating a secondary replica capable of processing read operations, the secondary replica is a read-only replica in which data of the secondary replica is synchronized with data in the primary replica;
• operating a delta replica providing quorum support without providing read or write operations, the delta replica comprising a copy of transactional log information of the primary replica that indicates changes that have been made to the data of the primary replica without including the data of the primary replica;
• and
• maintaining backup data synchronized with the data of the primary replica in a memory of the data processing system allocated to the tenant on a separate server from computing resources allocated to the plurality of replicas;
• automatically determining that the primary replica and the secondary replica have experienced a failure and are non-operational and that the delta replica is operational;
• and
• automatically instantiating a recovery workflow using the delta replica, the recovery workflow comprising:
• initializing a second secondary replica to be synchronized with a state of the data of the primary replica prior to the failure of the primary replica;
• causing the second secondary replica to download the backup data from the memory in the data processing system allocated to the tenant;
• updating the second secondary replica using the backup data to synchronize the second secondary replica with the state of the primary replica prior to the failure of the primary replica;
• and
• promoting the second secondary replica to become a new primary replica.

A data processing system comprising:

• a processor;
• and
• a machine-readable medium storing executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
• operating a database including a plurality of replicas associated with a tenant of the data processing system, the database comprises software for collecting, organizing, storing, accessing, and manipulating electronic data, the database including data structures for organizing and storing the electronic data, wherein a replica of the database represents a copy of the database, and wherein operating the database includes:
• operating a primary replica capable of processing both read and write operations;
• operating a secondary replica capable of processing read operations, the secondary replica is a read-only replica in which data of the secondary replica is synchronized with data in the primary replica;
• operating a delta replica providing quorum support without providing read or write operations, the delta replica comprising a copy of transactional log information of the primary replica that indicates changes that have been made to the data of the primary replica without including the data of the primary replica;
• and
• maintaining backup data synchronized with the data of the primary replica in a memory of the data processing system allocated to the tenant on a separate server from computing resources allocated to the plurality of replicas;
• automatically determining that a quorum loss has occurred in which the delta replica is operational and the primary replica and the secondary replica are non-operational;
• and
• automatically instantiating a recovery workflow using the delta replica, the recovery workflow comprising:
• initializing a second secondary replica to be synchronized with a state of the data of the primary replica prior to a failure of the primary replica;
• causing the second secondary replica to download the backup data from the memory in the data processing system allocated to the tenant;
• updating the second secondary replica using the backup data to synchronize the second secondary replica with the state of the primary replica prior to the failure of the primary replica;
• and
• promoting the second secondary replica to become a new primary replica.