Techniques for asynchronously pushing metadata in bulk

An apparatus for data management, comprising:

• at least one processor;
• at least one memory coupled with the at least one processor;
• and
• instructions stored in the at least one memory and executable by the at least one processor to cause the apparatus to:
• determine, by a first application in a destination computing environment of a data management system, that a bulk-push criterion is satisfied for a real-time data stream between the first application and a second application in a source computing environment of the data management system, wherein the bulk-push criterion is based at least in part on a capability of the first application and the second application to process a backlog associated with the real-time data stream;
• transmit, to an asynchronous metadata service associated with the data management system, a request including a first identifier of the second application for which the bulk-push criterion is satisfied and a second identifier of the real-time data stream, wherein the request is configured to cause the asynchronous metadata service to:
• query, based at least in part on the first identifier and the second identifier, a database in the source computing environment to identify a latest version of one or more rows that include metadata associated with the real-time data stream for which the bulk-push criterion is satisfied;
• and
• generate one or more data records comprising the latest version of the one or more rows that include the metadata associated with the real-time data stream for which the bulk-push criterion is satisfied;
• and
• receive, via an asynchronous data stream between the first application in the destination computing environment and the second application in the source computing environment, the one or more data records generated by the asynchronous metadata service.

A non-transitory computer-readable medium storing code for data management, the code comprising instructions executable by at least one processor to:

• determine, by a first application in a destination computing environment of a data management system, that a bulk-push criterion is satisfied for a real-time data stream between the first application and a second application in a source computing environment of the data management system, wherein the bulk-push criterion is based at least in part on a capability of the first application and the second application to process a backlog associated with the real-time data stream;
• transmit, to an asynchronous metadata service associated with the data management system, a request including a first identifier of the second application for which the bulk-push criterion is satisfied and a second identifier of the real-time data stream, wherein the request is configured to cause the asynchronous metadata service to:
• query, based at least in part on the first identifier and the second identifier, a database in the source computing environment to identify a latest version of one or more rows that include metadata associated with the real-time data stream for which the bulk-push criterion is satisfied;
• and
• generate one or more data records comprising the latest version of the one or more rows that include the metadata associated with the real-time data stream for which the bulk-push criterion is satisfied;
• and
• receive, via an asynchronous data stream between the first application in the destination computing environment and the second application in the source computing environment, the one or more data records generated by the asynchronous metadata service.