Big data distributed processing and secure data transferring with fallback control

A computer-implemented method of fallback control during in-flight data masking and on-demand encryption of big data on a network comprising the steps of:

• authenticating, by an obfuscation computer machine, a request for a big data dataset based on credentials received from a source,
• said obfuscation computer machine coupled to the network,
• said obfuscation computer machine containing computer-readable memory,
• said computer-readable memory including a list of executors available for processing the big data dataset, said list of available executors stored in a first sector of the computer-readable memory;
• said request stored in a second sector of the computer-readable memory;
• said big data dataset stored in a data store coupled to the network, said big data dataset including sensitive information and non-sensitive information;
• auto provisioning, by the obfuscation computer machine, a profile corresponding to the request, said profile stored in a third sector of the computer-readable memory;
• assigning, by the obfuscation computer machine, access rights for the request based on the profile, said access rights stored in a fourth sector of the computer-readable memory;
• defining, by the obfuscation computer machine, server configuration and data connection properties for the data store containing the big data dataset, said server configuration and said data connection properties stored in a fifth sector of computer-readable memory;
• establishing, by the obfuscation computer machine, a secure connection from the obfuscation computer machine to the data store based on the server connection and the data connection properties;
• registering, by the obfuscation computer machine, at least one multiple data obfuscation type stored in a sixth sector of computer-readable memory;
• loading, by the obfuscation computer machine into a seventh sector of computer-readable memory, the big data dataset from the data store on the network;
• searching, by the obfuscation computer machine, the big data dataset for the sensitive information to be redacted;
• redacting into a sanitized dataset, by the obfuscation computer machine, the sensitive information in the big data dataset based on said at least one multiple data obfuscation type and the non-sensitive information;
• storing, by the obfuscation computer machine into an eighth sector of computer-readable memory, the sanitized dataset;
• storing, progressively by the obfuscation computer machine into a ninth sector of computer-readable memory, a last safe state point reached during processing of the big data dataset;
• detecting, by the obfuscation computer machine, a crashed executor that crashed during said processing of the big data dataset;
• removing, by the obfuscation computer machine, the crashed executor from the list of available executors to create a list of non-crashed executors, said list of non-crashed executors stored in the first sector of the computer-readable memory;
• resubmitting, by the obfuscation computer machine, an uncompleted portion of an uncompleted task assigned to the crashed executor to one of said non-crashed executors, said uncompleted portion being determined by the last safe state point reached, whereby only the uncompleted portion needs to be further processed;
• encrypting, by the obfuscation computer machine, the sanitized dataset into encrypted data, said encrypted data stored in a tenth sector of computer-readable memory;
• and
• transmitting, by the obfuscation computer machine, the encrypted data in response to the request.

A non-transitory computer-readable medium with computer-executable instructions stored thereon executed by a processor on a computer machine to perform fallback control during in-flight data masking and on-demand encryption of a big data dataset stored in a data store on a network, said big data dataset including sensitive information and non-sensitive information, said computer machine communicatively coupled to the network, comprising:

• authentication instructions to authenticate a request for the big data dataset based on credentials received from a source,
• auto provision instructions to identify a profile corresponding to the request, said profile stored in a first sector of the computer-readable medium;
• access instructions to assign access rights for the request based on the profile, said access rights stored in a second sector of the computer-readable medium;
• server configuration and data connection instructions to define connection properties for the data store containing the big data dataset, said server configuration and said data connection properties stored in a third sector of the computer-readable medium,
• secure connection instructions to establish a secure connection from the computer machine to the data store based on the server connection and the data connection properties;
• registration instructions to register at least one multiple data obfuscation type stored in a fourth sector of the computer-readable medium;
• load instructions to load the big data dataset from the data store on the network into a fifth sector of the computer-readable medium;
• search instructions to search the big data dataset for the sensitive information to be redacted;
• redaction instructions to redact into a sanitized dataset the sensitive information in the big data dataset based on said at least one multiple data obfuscation type and the non-sensitive information;
• diagnosing instructions to identify RAM requirements and a current RAM allocation for the computer machine;
• rebate instructions to rebate a portion of the current RAM allocation that exceeds the RAM requirements for the computer machine;
• detection instructions to detect a crashed executor;
• remove instructions to remove the crashed executor from a broad list of available executors to create a reduced list of available executors such that the crashed executor is no longer used;
• state point instructions to store a last safe state reached during processing;
• identification instructions to identify any incomplete tasks assigned to the crashed executor;
• incremental load instructions to identify any incomplete portion of any said incomplete task based on said last safe state reached;
• transfer instructions to transfer said incomplete portion to an available executor in the reduced list of available executors, whereby only the incomplete portion of the incomplete task needs to be further processed;
• storage instructions to store the sanitized dataset into a sixth sector of the computer-readable medium;
• encryption instructions to encrypt the sanitized dataset into encrypted data, said encrypted data stored in a seventh sector of the computer-readable medium;
• and
• transmission instructions to transmit the encrypted data in response to the request.