US 11,886,439 B1Granted

Asynchronous change data capture for direct external transmission

AssigneeAmazon Technologies, Inc.

InventorsSangyong Hwang, Aakash Ashwin Shah, Yongsik Yoon, Kamal Kant Gupta, Saileshwar Krishnamurthy

Life of this patent

Estimating the term…

Loading patent…

Abstract

Change data capture is performed asynchronously for direct transfer to a stream processor. When updates to a database as part of database transactions are performed, a transaction log is generated that describes the database transactions. The transaction log may be evaluated to determine indications of changes included in the database transactions. The indications of the changes may be stored in memory according to the database transactions that include the indications of the changes. When a database transaction is determined to be committed to the database, the change indications may be sent directly from the memory to the stream processor.

Claims20

Dependent claims

A system, comprising:

a plurality of head nodes collectively implementing a database system, wherein individual ones of the plurality of head nodes respectively comprise:
at least one processor;
and
a memory;
and
wherein the database system is configured to:
perform, by the plurality of head nodes, a plurality of different database transactions that update a single database, wherein to perform the plurality of different database transactions, individual ones of the plurality of head nodes of the database system are configured to generate respective transaction logs that describes changes to the single database caused by the performing of the plurality of different database transactions by the plurality of head nodes;
and
generate, based at least in part on the generated transaction logs and asynchronous to the performance of the database transactions, a data stream of change indications, wherein to generate the data stream of change indications the database system is configured to:
evaluate, by respective ones of the plurality of head nodes, the respective transaction logs to generate indications of the changes to the single database;
group, in the memory of a primary head node of the plurality of head nodes according to the different database transactions, respective ones of the indications of the changes generated by the plurality of head nodes to generate a plurality of grouped indications of the changes to the single database, wherein individual ones of the plurality of grouped indications of the changes correspond to respective ones of the different database transactions performed by respective head nodes of the plurality of head nodes;
and
after a determination that one or more of the different database transactions are committed to the single database, send, from the memory of the primary head node directly to a stream processor external to the single database, the respective grouped indications of the changes of the committed one or more of the different database transactions performed by the plurality of head nodes.

10 more elements

A method, comprising:

asynchronous to a plurality of updates performed at a single database by two or more head nodes of the database:
evaluating respective logs of the two or more head nodes that describe the plurality of updates performed at the database for different database transactions to generate indications of changes to the single database performed by respective head nodes of the two or more head nodes;
storing the indications of the changes generated from the respective logs of the two or more head nodes in a memory of a change streaming manager, grouped according to the different database transactions;
and
after determining that one or more of the different database transactions are committed to the database, sending, from the memory of the change streaming manager to a stream processor external to the single database, the respective grouped indications of the changes of the committed one or more database transactions performed by the two or more head nodes.

2 more elements

One or more non-transitory, computer-readable storage media, storing program instructions that when executed on or across one or more computing devices cause the one or more computing devices to implement a single database that implements:

asynchronous to a plurality of updates performed at the single database by two or more head nodes of the single database:
evaluating respective logs of the two or more head nodes that describe the plurality of updates performed at the single database for different database transactions to generate indications of changes to the database performed by the two or more head nodes;
grouping, in a memory of a primary head node of the two or more head nodes according to the different database transactions, respective ones of the indications of the changes generated from the respective logs of the two or more head nodes to generate a plurality of grouped indications of the changes to the single database, wherein individual ones of the plurality of grouped indications of the changes correspond to respective ones of the different database transactions performed by respective head nodes of the two or more head nodes;
and
after determining that one or more of the different database transactions are committed to the single database, sending, from the memory of the primary head node directly to a stream processor external to the single database, the respective grouped indications of the changes of the committed one or more database transactions performed by the two or more head nodes.

2 more elements

Dependencies are inferred from the claim text.