Systems and methods for securely managing data in distributed systems

A distributed data analytics system, comprising:

• persistent storage resources configured to store imported data pertaining to respective entities;
• a content service configured to provide access to data analytics functionality pertaining to the imported data of the respective entities stored within the persistent storage resources;
• an import engine configured to acquire datasets pertaining to the respective entities from one or more data management systems;
• and
• a data manager configured to secure the imported data within the persistent storage resources, the security manager configured to:
• obtain data keys for use in encrypting datasets comprising imported data pertaining to the respective entities for storage within the persistent storage resources;
• retain secure key data associated with the data keys in the persistent storage resources, the secure key data corresponding to master keys of the respective entities, wherein access to the master keys is controlled by the respective entities;
• and
• service requests to access imported data pertaining to the respective entities from the content service, wherein servicing a first request to access imported data pertaining to a particular entity stored within a first encrypted dataset comprises:
• retrieving first secure key data corresponding to a first data key required to decrypt the encrypted dataset, the first secure key data corresponding to a first master key,
• attempting to access the first data key from a key manager by use of the retained secure key data, the key manager configured to provide access to the first data key in response to determining that the particular entity has authorized the data analytics system to access the first master key,
• returning a first decrypted dataset comprising the requested imported data pertaining to the particular entity in response to accessing the first data key from the key manager, and
• returning an error indication in response to failing to access the first data key.

A non-transitory computer-readable storage medium comprising instructions configured to cause a computing device to perform a method for securely managing data pertaining to a plurality of different entities within a distributed platform, comprising:

• importing datasets from one or more data management systems, each imported dataset comprising unencrypted data pertaining to respective entities of the plurality of different entities;
• and
• controlling access to the imported datasets within the distributed platform in accordance with access rights controlled by the respective entities, comprising:
• generating encrypted datasets from the imported datasets using data encryption keys acquired from a key management system, the data encryption keys used to encrypt the imported datasets of the respective entities corresponding to master keys having access rights controlled by the respective entities and secure key data derived from the master keys;
• retaining the encrypted datasets and corresponding secure key data in persistent storage of the distributed platform;
• excluding the unencrypted data of the imported datasets and the data encryption keys from the persistent storage of the distributed system;
• servicing requests to access unencrypted data pertaining to the respective entities within the distributed platform, wherein servicing a request to access unencrypted data pertaining to a first entity comprises:
• retrieving a first encrypted dataset comprising the requested unencrypted data from the persistent storage,
• requesting that the key management system generate a data encryption key required to decrypt the first encrypted dataset by use of secure key data corresponding to the first encrypted dataset and a specified master key having access rights controlled by the first entity,
• granting access to the requested unencrypted data pertaining to the first entity in response to the key management system providing the required data encryption key in response to the requesting, and
• denying access to the requested unencrypted data pertaining to the first entity in response to the key management system refusing to provide the required data encryption key in response to the requesting.