Out-of-band schema tracking for change-data-capture logs
AssigneeAmazon Technologies, Inc.
InventorsKanishka Chaturvedi, Andrew Willis Callahan, Kunal Anil Lillaney, Punit Rajgaria, Md Arif
Life of this patent
Estimating the term…
Abstract
Methods for replicating transactional tables of a transactional database to a recipient system, such as an analytical database, and maintaining updates to those transactional table representations are disclosed. More particularly, change-data-capture items are generated by a transport engine, that is separate from a database engine for the transactional database. Also, schema data is maintained out-of-band from the database engine and is stitched with committed change payload contents by the transport engine.
Claims20
A system, comprising:
- one or more computing devices, wherein the one or more computing devices are configured to:
- implement a transactional database;
- maintain, at the transactional database, a database table comprising data items;
- and
- write, at the transactional database, changes made to the database table to a change-data-capture log, wherein writing the changes to the change-data-capture log comprises:
- writing transaction identifiers for the changes to a transaction table;
- writing payload contents of the changes to one or more change log contents tables;
- and
- maintaining a schema table comprising schema definitions for schemas to be applied to the payload contents to represent the changes;
- and
- one or more computing devices configured to implement a transport mechanism, wherein the transport mechanism is configured to:
- read committed changes from the transaction table;
- generate change-data-capture items for the committed changes, wherein generating the change-data-capture items comprises:
- reading, for a given committed change, payload contents for the given change from the one or more change log contents tables;
- and
- applying a corresponding schema definition from the schema table to the payload contents read from the one or more change log contents tables, wherein the schema definition to be applied is identified based on an identifier associated with the change and a corresponding identifier included in the schema table;
- and
- provide the change-data-capture items.
A method comprising:
- maintaining a database table comprising data items;
- receiving transactions indicating changes to be made to the database table;
- updating the database table to reflect the changes;
- and
- writing the changes made to the database table to a change-data-capture log, wherein writing the changes to the change-data-capture log comprises:
- writing transaction identifiers for changes to a transaction table;
- writing payload contents of the changes to one or more change log contents tables;
- and
- maintaining a schema table comprising schema definitions for schemas to be applied to the payload contents to represent the changes.
One or more non-transitory, computer-readable, storage media storing program instructions that, when executed on or across one or more processors, cause the one or more processors to:
- read committed transactions from a transaction table;
- generate change-data-capture items for transport, wherein said generating the change-data-capture items comprises:
- reading, for a given committed change, payload contents for the given change from one or more change log contents tables;
- and
- applying a corresponding schema definition from a schema table to the payload contents read from the one or more change log contents tables, wherein the schema definition to be applied is identified based on an identifier associated with the change and a corresponding identifier included in the schema table;
- and
- provide the change-data-capture items for transport to a recipient system.
Dependencies are inferred from the claim text.