Method of queue design for data storage and management

Queue design method for a queue data storage and management including the steps of:

• step 1:
• build deployment models;
• simultaneously deploy a set of many distributed nodes, install a database using a RAM synchronization mechanism for these nodes;
• in all distributed nodes, wherein only one node can do business with the database at a time, the rest of the nodes have data redundancy function;
• step 2:
• initialize values when the method first launches;
• an input of this step is an output of step 1, that is, a parameter to configure a list of synchronous nodes, in addition, the input of this step is an amount of storage allocated for the queue, a threshold value of a number of elements and a storage time threshold for an element;
• At this step, perform initialization of values comprising:
• a queue table and an index management table;
• a storage value for the queue corresponding to a maximum number of elements allowed to be stored in the queue table;
• in case the queue reaches an element maximum, new elements will be overwritten from a beginning of the queue;
• the threshold value of a number of elements corresponding to a maximum number of elements in a data block to be processed;
• wherein this value depends on the business requirements of each system;
• according to the actual evaluation where applicable, the threshold value of the elements is calculated by taking an integer value of the [queue storage capacity value]/k, where k is a factor of safety (10<k<storage capacity value for the queue);
• a time threshold value corresponding to a maximum storage time of a data element in the queue;
• a queue check cycle value, which depends on a speed at which data is written to the queue;
• monitoring and handling events that exceed the threshold of the queue;
• setting the test cycle conditions through the queue test cycle value value;
• initialize the last processing time value to the current time and store it in the state of the process;
• at the end of this step, obtain three main components constituting a queue:
• the index management table, the queue table and the monitoring and data processing process for the queue;
• step 3:
• handle writing data to the queue;
• input is data from another business or system sent to the queue;
• At this step, perform the following steps:
• when data is sent in, read from the last index value from index management table;
• insert element {last index value, data} into queue table;
• update the new last index value according to the increment and cycle rule between 0 and the storage value for the queue−1;
• update the new last index value to the index management table;
• then compute the temporary last index value (Last_tmp) as the result of the calculation (last index value+1), comparing Last_tmp with the storage value for the queue;
• if Last_tmp≥storage threshold value for the queue, update {last index value, 0} to the index management table;
• in contrast, update {last index, Last_tmp} to the index management table;
• at the end of this step, the output is the queue table containing the data just written;
• step 4:
• detect when the threshold is exceeded and process data in the queue;
• the input to this step is the data in the queue written in step 3;
• at this step, perform periodic checks for over-threshold conditions at the queue monitoring process;
• at the end of this step, a block of data is obtained according to either the number of elements threshold or the time threshold;
• step 5:
• remove processed data from the queue;
• the input of this step is the first and last index values of the elements just processed in step 4;
• at this step, initiate a new process and perform the following actions:
• if last index value=first index value, terminate the process;
• if last index value>first index value, delete the elements with the key incrementing from the first index value to the last index value−1 from the queue table;
• if last index value<first index value, delete elements with keys incrementing from the first index value to the storage value for the queue−1 and from 0 to the last index value−1 from the queue table;
• at the end of this step, the queue table has been cleaned up;
• step 6:
• monitor the queue and early warn;
• the input to this step is the queue table information initialized from step 2, at this step, perform a process initialization that periodically checks the number of elements in the queue, doing:
• check the number of elements stored in the queue table, if the number of elements in the queue table exceeds 70% of the storage value for the queue, perform logging or broadcast a warning message;
• set a timer after a certain amount of time, repeat the check for the number of elements stored in the queue table;
• the end of this step is a warning if any;
• the end result is to build a queue that meets the basic functions of storing data, grouping data according to one of two criteria:
• exceeding the threshold of number of elements and exceeding the threshold of storage time;
• especially, the queue is capable of withstanding large loads, storing large data and supporting data redundancy, minimizing the possibility of data loss in the event of a failure;
• in addition, the allocation of fixed memory for the queue ensures safety for the system and neighboring systems.