Data pipeline architecture
AssigneeJPMorgan Chase Bank, N.A.
InventorsRalph Joseph PINHEIRO, Olutayo IBIKUNLE
Life of this patent
Estimating the term…
Abstract
A method and a system for managing data flows and data processing operations with respect to a platform are provided. The method includes: executing a first set of microservice applications for coordinating data flows with respect to a data repository; executing a second set of microservice applications for coordinating data processing operations; and executing a third set of microservice applications for performing metadata processing that relates to the data flows and the data processing operations. The system provides an architecture that is designed for delivering information with speed, scale, and quality to diverse destinations and use cases and providing advanced data processing to support real-time streaming processes and aggregated batch processes.
Claims17
A method for managing data flows and data processing operations with respect to a platform, the method being implemented by at least one processor, the method comprising:
- executing, by the at least one processor, a first set of microservice applications for coordinating data flows with respect to a first memory;
- executing, by the at least one processor, a second set of microservice applications for coordinating data processing operations;
- and
- executing, by the at least one processor, a third set of microservice applications for performing metadata processing that relates to the data flows and the data processing operations,
- wherein the first set of microservice applications includes a data flow order microservice application that relates to determining a processing order for a cluster-specific set of data flows that are assigned to a corresponding cluster, and
- wherein the method is implemented in a distributed hybrid cloud computing environment across a plurality of data planes, and
- wherein the method further comprises:
- performing predictive workload balancing among the plurality of data planes based on current and predicted demand;
- determining resource availability among the plurality of data planes;
- ensuring compliance with at least one data privacy policy;
- and
- Performing tracking of at least one predetermined service level agreement (SLA) with respect to a current backlog.
(canceled)
(canceled)
(canceled)
A computing apparatus for managing data flows and data processing operations with respect to a platform, the computing apparatus comprising:
- a processor;
- a memory;
- and
- a communication interface coupled to each of the processor and the memory,
- wherein the processor is configured to:
- execute a first set of microservice applications for coordinating data flows with respect to the memory;
- execute a second set of microservice applications for coordinating data processing operations;
- and
- execute a third set of microservice applications for performing metadata processing that relates to the data flows and the data processing operations,
- wherein the first set of microservice applications includes a data flow order microservice application that relates to determining a processing order for a cluster-specific set of data flows that are assigned to a corresponding cluster, and
- wherein the second set of microservice applications includes at least one from among a data ingestion microservice application that relates to ingesting data from a source that is external to the platform, a data stream processing microservice application that relates to performing continuous stream processing, and a data movement microservice application that relates to a bulk-copying of data from a first location within the platform to a second location within the platform.
(canceled)
(canceled)
Dependencies are inferred from the claim text.