System and method for cloud-based replication of data

A system for data replication comprising:

• a processor;
• a memory in communication with the processor, wherein the memory comprises:
• a virtual table definition, the virtual table definition comprising:
• a definition of a table schema that includes virtual table fields and data types of the virtual table fields;
• and
• a mapping of the virtual table fields to target data types of a cloud hosted database;
• and
• a data collection definition that defines data of interest from a data source;
• a transformation engine that comprises instructions executable for:
• creating a virtual table according to the virtual table definition, the virtual table comprising the virtual table fields;
• and
• populating the virtual table fields with the data of interest according to the data collection definition;
• and
• a cloud connector, the cloud connector comprising instructions executable for:
• sending virtual table information and virtual table records to a cloud environment for replication of the data of interest in the cloud hosted database when the table schema does not match a pre-existing schema in the cloud hosted database or in the heterogeneous on-premises data sources, the virtual table information comprising:
• the table schema;
• and
• the mapping of the virtual table fields to the target data types;
• creating the physical table in the cloud hosted database, wherein a structure of the physical table for replication is defined by the table schema and the mapping of the virtual table fields to the target data types received in the virtual table information.

A system for data replication comprising:

• an on-premises system, the on-premises system comprising:
• an on-premises system processor;
• an on-premises system memory in communication with the on-premises system processor, wherein the on-premises system memory comprises:
• a virtual table definition, the virtual table definition comprising:
• a definition of a table schema that includes virtual table fields and data types of the virtual table fields;
• and
• a mapping of the virtual table fields to target data types of a cloud hosted database;
• and
• a data collection definition that defines data of interest from a data source;
• a transformation engine that comprises instructions executable for:
• creating a virtual table according to the virtual table definition, the virtual table comprising the virtual table fields;
• and
• populating the virtual table fields with the data of interest according to the data collection definition;
• and
• a cloud connector, the cloud connector comprising instructions executable for:
• sending virtual table information and virtual table records to a cloud system for replication of the data of interest in the cloud hosted database, the virtual table information comprising:
• the table schema;
• and
• the mapping of the virtual table fields to the target data types;
• and
• the cloud system, the cloud system comprising:
• the cloud hosted database;
• a cloud system processor;
• a cloud system memory in communication with the cloud system processor, wherein the cloud system memory comprises:
• a queue;
• a message producer, the message producer comprising instructions executable for:
• receiving the virtual table information and the virtual table records;
• storing the virtual table information and the virtual table records;
• writing references to the virtual table information and the virtual table records to the queue;
• a message consumer, the message consumer comprising instructions executable for:
• dequeuing the references to the virtual table information and the virtual table records from the queue;
• based on the references to the virtual table information and the virtual table records, accessing the virtual table information and the virtual table information;
• and
• writing the virtual table records to a physical table that corresponds to the virtual table to replicate the data of interest from the on-premises system to the cloud hosted database, and wherein at least one of the message producer or the message consumer comprises instructions executable for:
• determining if the physical table that corresponds to the virtual table exists in the cloud hosted database;
• and
• based on a determination that the physical table that corresponds to the virtual table does not exist in the cloud hosted database, creating the physical table in the cloud hosted database, wherein a structure of the physical table is defined by the table schema and the mapping of the virtual table fields to the target data types received in the virtual table information.