System and method for achieving optimal change data capture (cdc) on hadoop
AssigneeJPMorgan Chase Bank, N.A.
InventorsJagmohan SINGH, Bharaneedaran SARAVANAN, Prasad V. PONDICHERRY
Life of this patent
Estimating the term…
Abstract
An embodiment of the present invention is directed to configuring optimal parallelism on Hadoop processes at specific processing points for improved performance and throughput. An embodiment of the present invention forces Hadoop into high parallelism independent of a HDFS file size. An embodiment of the present invention achieves an order of magnitude improvement in performance and data throughputs to meet the SLAs. An embodiment of the present invention provides a design pattern unique to distributed processing in Hadoop to accomplish the required data throughputs for the CDC process.
Claims20
A system that processes optimal change data capture on a Hadoop cluster of data nodes, the system comprising:
- an application component that receives change data;
- a memory component that stores change data;
- and
- a computer server coupled to the application component and the memory, the computer server comprising a programmed computer processor configured to perform the steps of:
- receiving a change data capture file that represents change activity for a current time period with respect to the application component;
- receiving a previous full data snapshot file;
- applying a data quality check process on the change data capture file, using a forced parallelism configuration that applies processing on a full set of Hadoop data nodes;
- applying change data capture records, using the forced parallelism configuration that applies processing on the full set of Hadoop data nodes;
- and
- generating a full data output based on the change data capture records.
A method that processes optimal change data capture on a Hadoop cluster of data nodes, the method comprising the steps of:
- receiving a change data capture file that represents change activity for a current time period with respect to an application component;
- receiving a previous full data snapshot file;
- applying a data quality check process on the change data capture file, using a forced parallelism configuration that applies processing on a full set of Hadoop data nodes;
- applying change data capture records, using the forced parallelism configuration that applies processing on the full set of Hadoop data nodes;
- and
- generating a full data output based on the change data capture records.
Dependencies are inferred from the claim text.