Resumable and Online Schema Transformations

In a database computing environment, a method of generating a new data structure, the method comprising:

• receiving a database command from a user for modifying an existing data structure or creating a new data structure;
• constructing a query that will execute the database commend and providing the query to a query optimizer component;
• at the query optimizer component, transforming the query into a first query execution plan, wherein the first query execution plan comprises one or more operations that persist operation state of execution of the first query plan;
• receiving at least a portion of data specified in the query and constructing at least a portion of the new data structure by executing at least a portion of the first query plan;
• while receiving the at least a portion of data specified in the query and constructing a least a portion of the new data structure, persisting operation state of execution of the first query plan;
• determining that an interruption of execution of the first query plan has occurred;
• as a result of determining that an interruption of execution of the first query plan has occurred, generating an updated query plan using the persisted operation state of execution of the first query plan, such that the new query plan can be executed without needing to completely restart receiving data and constructing the new data structure;
• resuming receiving data and constructing the new data structure by executing the updated query plan;
• receiving external updates to the new data structure either while data is being received as a result of execution of the first query plan or while execution of the first query plan was interrupted;
• and
• transactionally validating the external updates in conjunction with executing the updated query plan.

A database computer system comprising:

• an execution engine, wherein the execution engine is configured to perform at least the following:
• receive a database command from a user for modifying an existing data structure or creating a new data structure;
• construct a query that will execute the database command and transform the query into a first query execution plan, wherein the first query execution plan comprises one or more operations that persist operation state of execution of the first query plan;
• provide the first execution plan to a query processor;
• the query processor, wherein the query processor is configured to perform at least the following;
• receive at least a portion of data specified in the query and constructing at least a portion of the new data structure by executing at least a portion of the first query plan;
• while receiving the at least a portion of data specified in the query and constructing a least a portion of the new data structure, persist operation state of execution of the first query plan:
• determine that an interruption of execution of the first query plan has occurred;
• as a result of determining that an interruption of execution of the first query plan has occurred, generate an updated query plan using the persisted operation state of execution of the first query plan, such that the new query plan can be executed without needing to completely restart receiving data and constructing the new data structure;
• resume receiving data and constructing the new data structure by executing the updated query plan;
• and
• receive external updates to the new data structure either while data is being received as a result of execution of the first query plan or while execution of the first query plan was interrupted;
• and
• a storage engine, wherein the storage engine is configured to transactionally validate the external updates in conjunction with executing the updated query plan.