System and method for processing large amounts of transactional data

A method of processing large amounts of transactional data, comprising the steps of:

• generating at a configuration service system information related to configuration of a plurality of specific instances of a generic process to be performed on the transactional data, each of the specific instances comprising a plurality of processes;
• storing a first set of data used in the generic process at a reference data server;
• at one or more processing modules of a first stage processing system, for each specific instance, executing at least one first process of the plurality of processes that make up the specific instance on the transactional data based on a corresponding one of a plurality of instance specific configurations generated by the configuration service system;
• the step of executing comprising:
• at each of the one or more processing modules of the first stage processing system, storing a second set of data used in the at least one first process at an in-memory cache;
• and
• at each of the one or more processing modules, re-initializing one or more first task executors based on the corresponding one of the plurality of instance specific configurations to perform one or more tasks associated with the at least one first process so as to generate first stage processed transactional data;
• at one or more processing modules of a second stage processing system, for each specific instance, executing at least one second process of the plurality of processes that make up the specific instance on the first stage processed transactional data based on a corresponding one of a plurality of instance specific configurations generated by the configuration service system;
• the step of executing comprising:
• at each of the one or more processing modules of the second stage processing system, storing a third set of data used in the at least one second process at an in-memory cache;
• and
• at each of the one or more processing modules, re-initializing one or more first task executors based on the corresponding one of the plurality of instance specific configurations to perform one or more tasks associated with the at least one second, process so as to generate second stage processed transactional data;
• managing using a flow controller flow of data between the first and second stage processing systems;
• and
• distributing using a load balancer processing load within each of the first and second stage processing systems.

A computer system comprising one or more processors, one or more communication devices operatively connected to the one or more processors, and one or more computer-readable media containing computer-readable instructions executable on the one or more processors, the instructions being related to performance of a method for processing large amounts of transactional data, comprising the steps of:

• generating at a configuration service system information related to configuration of a plurality of specific instances of a generic process to be performed on the transactional data, each of the specific instances comprising a plurality of processes;
• storing a first set of data used in the generic process at a reference data server;
• at one or more processing modules of a first stage processing system, for each specific instance, executing at least one first process of the plurality of processes that make up the specific instance on the transactional data based on a corresponding one of a plurality of instance specific configurations generated by the configuration service system;
• the step of executing comprising:
• at each of the one or more processing modules of the first stage processing system, storing a second set of data used in the at least one first process at an in-memory cache;
• and
• at each of the one or more processing modules, re-initializing one or more first task executors based on the corresponding one of the plurality of instance specific configurations to perform one or more tasks associated with the at least one first process so as to generate first stage processed transactional data;
• at one or more processing modules of a second stage processing system, for each specific instance, executing at least one second process of the plurality of processes that make up the specific instance on the first stage processed transactional data based on a corresponding one of a plurality of instance specific configurations generated by the configuration service system;
• the step of executing comprising:
• at each of the one or more processing modules of the second stage processing system, storing a third set of data used in the at least one second process at an in-memory cache;
• and
• at each of the one or more processing modules, re-initializing one or more first task executors based on the corresponding one of the plurality of instance specific configurations to perform one or more tasks associated with the at least one second process so as to generate second stage processed transactional data;
• managing using a flow controller flow of data between the first and second stage processing systems;
• and
• distributing using a load balancer processing load within each of the first and second stage processing systems.