Managing extract, transform and load systems

A system for hosting an extract, transform and load, ETL, method with extract, transform and load phases, the system comprising:

• a source system operable to supply units of data in an extract phase;
• a target system operable to receive units of data in a load phase;
• and
• an extract, transform and load engine comprising:
• an extract engine operable to perform the extract phase by extracting units of data from the source system;
• a queue configured to receive units of data from the extract engine;
• a processing engine operable to perform the transform and load phases by receiving units of data from the queue, transforming the units of data and loading the units of data to the target system;
• a pace manager operable to monitor the queue, wherein, if the pace manager detects that there are more than a threshold number of requeued units of data in the queue, then the pace manager is operable to impose at least one restriction on the extract phase;
• and
• the pace manager being further operable to monitor a history of units of data held in the queue and to delete one or more of the units of data based on their history, respectively, when the units of data fail a threshold number of times.

An extract, transform and load (ETL) method, the method comprising:

• an extract phase of extracting units of data from a source system and placing them in a queue;
• a transform phase of receiving units of data from the queue and processing them;
• and
• a load phase of loading units of data that have been processed to a target system, wherein the method:
• requeues units of data that are subject to a rate limit error in the load phase;
• monitors the queue to detect requeued units of data therein and, if there are more than a threshold number of requeued units of data in the queue, imposes at least one restriction on the extract phase;
• and
• monitors a history of units of data held in the queue and deletes one or more of the units of data based on their history, respectively, when the units of data fail a threshold number of times.