Multi-substrate fault tolerant continuous delivery of datacenter builds on cloud computing platforms

A non-transitory computer readable storage medium for storing instructions that when executed by one or more computer processors cause the one or more computer processors to perform steps for configuring data centers on a cloud platform, the steps comprising:

• configuring on the cloud platform, a primary control datacenter and a secondary control datacenter;
• receiving a declarative specification for configuring a tenant datacenter on the cloud platform, the tenant datacenter associated with a tenant of a multi-tenant system, the declarative specification describing a hierarchy of datacenter entities, wherein one or more datacenter entities of the hierarchy represent service groups and one or more other datacenter entities of the hierarchy represent services for execution on the cloud platform;
• sending instructions to the primary control datacenter to configure the tenant datacenter, the instructions causing the primary control datacenter to perform steps comprising:
• configuring an infrastructure for the tenant datacenter on the cloud platform, the infrastructure configured for a particular service represented by a datacenter entity of the hierarchy of datacenter entities specified by the declarative specification, and
• periodically transmitting a state of the infrastructure configured for the tenant datacenter to the secondary control datacenter;
• receiving request to transfer control of configuration of the tenant datacenter from the primary control datacenter to the secondary control datacenter;
• and
• responsive to receiving the request to transfer control, redirecting requests associated with configuration of the tenant datacenter from the primary control datacenter to the secondary control datacenter.

A computing system comprising:

• one or more computer processors;
• and
• a non-transitory computer readable storage medium for storing instructions that when executed by the one or more computer processors cause the one or more computer processors to perform steps for configuring data centers on a cloud platform, the steps comprising:
• configuring on the cloud platform, a primary control datacenter and a secondary control datacenter;
• receiving a declarative specification for configuring a tenant datacenter on the cloud platform, the tenant datacenter associated with a tenant of a multi-tenant system, the declarative specification describing a hierarchy of datacenter entities, wherein one or more datacenter entities of the hierarchy represent service groups and one or more other datacenter entities of the hierarchy represent services for execution on the cloud platform;
• sending instructions to the primary control datacenter to configure the tenant datacenter, the instructions causing the primary control datacenter to perform steps comprising:
• configuring an infrastructure for the tenant datacenter on the cloud platform, the infrastructure configured for a particular service represented by a datacenter entity of the hierarchy of datacenter entities specified by the declarative specification, and
• periodically transmitting a state of the infrastructure configured for the tenant datacenter to the secondary control datacenter;
• receiving a request to transfer control of configuration of the tenant datacenter from the primary control datacenter to the secondary control datacenter;
• and
• responsive to receiving the request to transfer control, redirecting requests associated with configuration of the tenant datacenter from the primary control datacenter to the secondary control datacenter.