Medical ETL task dispatching method, system and apparatus based on multiple centers

A medical ETL task dispatching method based on multiple centers, comprising steps of:

• step S1:
• generating ETL tasks, collecting resource demands of the ETL tasks and determining a time prediction equation by using a test machine, and testing and verifying the ETL tasks;
• step S11:
• generating the ETL tasks, operating the ETL tasks through the test machine, dividing data in an ETL task operating process into test data and verification data, and collecting resource demands of the test data and resource demands of the verification data, respectively;
• step S12:
• reading a data volume and a data reading rate of the test data by using the test machine, and determining the time prediction equation PRE_Tik according to the data volume and the data reading rate;
• PRE_T
• i
• k
• =
• a
• ⁢
• V
• k
• I
• k
• +
• b
• wherein Vk represents a data volume, to be processed, of an ETL task Taski in a stage k, Ik represents the data reading rate, and a and b are constant indexes;
• step S13:
• obtaining prediction time of the ETL tasks corresponding to the test data by using the time prediction equation;
• step S14:
• verifying the resource demands and the prediction time, and when the resource demands of the test data meet the resource demands of the verification data, and meanwhile, a difference value between the prediction time and actual execution time of the ETL tasks corresponding to the verification data is less than a preset threshold value, completing test and verification of the ETL tasks;
• step S2:
• deploying the ETL tasks to a hospital center, and dispatching, by the hospital center, the ETL tasks to a plurality of executors through a dispatching machine for execution;
• step S21:
• deploying the ETL tasks to the hospital center;
• step S22:
• determining the prediction time of the ETL tasks by using the time prediction equation, wherein step S22 comprises determining the prediction time of the ETL tasks, by using the time prediction equation, through the number of ETL tasks to be processed in a current stage and a data reading rate of the hospital center;
• step S23:
• determining a priority of the ETL tasks by using the prediction time, wherein a shortest task priority principle is used to determine the priority, and the shortest task priority principle stipulates that a ETL task with shorter remaining processing time in the current stage has a higher priority to reduce average waiting time of all ETL tasks;
• step S24:
• dispatching, by the dispatching machine, the ETL tasks to the executors for execution according to the priority of the ETL tasks;
• step S241:
• initiating, by the dispatching machine, active task queues and expired task queues;
• step S242:
• adding the ETL tasks to the active task queues according to the priority;
• step S243:
• when the ETL tasks in the active task queues are empty, exchanging the active task queues and the expired task queues, and continuously performing, by the dispatching machine, distributing and dispatching from the active task queues;
• step S3:
• collecting and counting up, by the dispatching machine, resource index vectors reported by each executor and resource demand vectors of ETL tasks to be dispatched in the current stage, and screening an executor set meeting resource demands of the ETL tasks to be dispatched;
• step S4:
• calculating a current task load of each executor in the executor set;
• step S5:
• selecting, by the dispatching machine, the executor with a minimum current task load to execute the ETL tasks according to the current task load of each executor;
• and
• step S6:
• adding, by the dispatching machine, the ETL tasks to executor active queues, determining the priority of the ETL tasks in the executor active queues according to prediction time determined by the prediction equation, and selecting, by the dispatching machine, the ETL tasks from the executor active queues according to the priority for execution;
• in the ETL task execution process, setting an ETL task operation time threshold value, when the ETL task execution time is greater than or equal to the ETL task operation time threshold value, pausing execution of the ETL tasks, adding the ETL tasks to the executor expire queues, and waiting for next-time dispatching;
• in the ETL task execution process, detecting ETL task stage information, when stages are switched, pausing execution of the ETL tasks, adding the ETL tasks to dispatching machine expired task queues, and waiting for re-dispatching by the dispatching machine;
• and
• in the ETL task execution process, when the executor active queues are empty after executor dispatching, exchanging the executor active queues and the executor expire queues, and continuously performing, by the dispatching machine, dispatching and execution from the executor active queues.