System and method for automatically managing storage resources of a big data platform

A computer implemented method for creating a predictive machine learning engine for predicting data storage usage for managing storage resources of a distributed file system, the method comprising the steps of:

• determining trends in received electronic past storage usage data for each directory of a plurality of directories of the distributed file system;
• obtaining a space quota limit for each said directory for imposing a maximum limit on total storage used in each said directory, the space quota limit having an associated expected future time being a maximum time for use of the storage resources of each said directory;
• predicting a projected storage usage data for each said directory extending for a future time period, including an expected storage usage amount for each said directory at the expected future time;
• wherein the projected storage usage data is computed as a function of both of:
• a computed first derivative of a curve interpolated of actual demand data fitting the past storage usage data for each said directory projected to at least the expected future time;
• and a computed first derivative of a moving average of the curve projected to the expected future time indicative of an average rate of change of the projected storage usage over time;
• wherein the first derivative of a moving average of the curve defines a first slope indicating the rate of change of the projected storage usage over time;
• and the first derivative of the moving average of the curve defines a second slope indicating the average rate of change of the projected storage usage over time;
• comparing the expected storage usage amount to the space quota limit for each said directory to determine whether a need for change of the space quota limit exists;
• computing a correction coefficient for each said directory based on aggregated projected need for each said directory relative to a total disk storage availability of the plurality of directories;
• adjusting the space quota limit to a new space quota limit based on the correction coefficient applied to the expected storage usage amount;
• and,
• generating the prediction machine learning engine for applying the new space quota limit to each said directory having the need for change;
• wherein the projected storage usage data for a first directory is computed as one of:
• a median and an average curve of the first derivative of the curve projected to at least the expected future time and the first derivative of the moving average of the curve projected to the expected future time and utilized to calculate the expected value.

A computer device for automatically managing storage resources of a distributed file system, the computer device comprising:

• a storage device storing instructions;
• a communications interface;
• at least one processor in communication with the storage device and the communications interface, the at least one processor configured to execute the instructions for:
• obtaining actual past storage usage data of a first directory from a plurality of directories of the distributed file system extending from a past time to a current time;
• detecting a space quota limit for the first directory, the space quota limit defining a maximum limit on total storage for the first directory and associated with a pre-defined expected future time defining a maximum time for use of resources of the first directory;
• determining projected storage usage data of the first directory representing a projected storage usage for the first directory over a future time period and as a function of both of:
• a first derivative of a curve representing the actual past storage usage data projected to at least the expected future time, the first derivative being a rate of change of the projected storage usage over time;
• and a first derivative of a moving average of the curve projected to at least the expected future time indicative of an average rate of change of the projected storage usage over time;
• wherein the first derivative of a moving average of the curve defines a first slope indicating the rate of change of the projected storage usage over time;
• and the first derivative of the moving average of the curve defines a second slope indicating the average rate of change of the projected storage usage over time;
• obtaining an aggregated correction coefficient providing an indication of aggregated projected storage usage needs of all other remaining distributed file system directories relative to the projected storage usage data of the first directory;
• and
• in response to determining an expected value of the projected storage usage data at the expected future time is inconsistent with the space quota limit, adjusting the space quota limit for the first directory to a new quota limit based on the expected value weighted by the aggregated correction coefficient;
• wherein the projected storage usage data for the first directory is computed as one of:
• a median and an average curve of the first derivative of the curve projected to at least the expected future time and the first derivative of the moving average of the curve projected to the expected future time and utilized to calculate the expected value.