Method and system for aggregating and managing data from disparate sources in consolidated storage

A distributed computing platform comprising:

• a first computer system that includes one or more data hardware processors coupled to a transceiver, wherein the transceiver is configured to:
• receive a first plurality of electronic data transaction messages from a plurality of different exchange computer systems, each one of the plurality of electronic data transaction messages including data fields that reflect how a corresponding one of the plurality of different exchange computer systems processed a corresponding electronic order submitted thereto, and
• receive a second plurality of electronic data messages from a plurality of different participant computer systems that each submit orders to different ones of the plurality of different exchange computer systems for execution thereby, wherein the first plurality of electronic data transaction messages and the second plurality of electronic data messages are in different data formats;
• wherein the one or more hardware processors is configured to:
• generate a combined first data set that is in JavaScript Object Notation (JSON) format by parsing securities transactions from the first plurality of electronic data transaction messages and the second plurality of electronic data messages to identify specific data fields for each security transaction, and then mapping those data fields to data elements of the combined first data set,
• partition the combined first data set into a plurality files, wherein orders that are part of the same order chain are determined to be partitioned into the same file,
• encrypt the plurality of files using a first encryption key, and
• transmit the encrypted plurality of files for storage on another computer system;
• a cloud-based computer system that includes a plurality of computing nodes, with each comprising hardware processing resources, each one of the plurality of computing nodes executing one or more virtual machines, the cloud-based computer system providing for dynamic scaling of the computing nodes that are allocated for handling processing and/or storage of data communicated from the first computer system;
• a secure storage system under control of at least one of the plurality of computing nodes of the cloud-based computer system, the cloud-based computer system comprising:
• a first data store configured to accept data that is written thereto, wherein once data is written to the first data store it can only be read, but not deleted or modified until being automatically deleted in response to expiration of a retention period,
• a plurality of participant data stores that are each respectively associated with a corresponding one of a plurality of participant accounts of a plurality of participants;
• and
• wherein the hardware processing resources of the cloud-based computer system are configured to:
• receive the transmitted encrypted plurality of files from the first computer system,
• write the encrypted plurality of files that are in JSON to the first data store, wherein the plurality of files that are written to the first data store are readable a plurality of times after the initial writing, but after the initial writing of the corresponding plurality of files cannot be deleted or modified until being automatically deleted in response to expiration of a retention period,
• receive a search query for data that is contained in the first data set that is spread across the plurality of files that are in JSON,
• execute the search query in parallel across multiple ones of the plurality of computing nodes and on each of those nodes decrypt a corresponding subset of the plurality of encrypted files that are stored to the first data store, wherein individual files of the plurality of encrypted files are assigned to the same computing node for processing, wherein the files are only decrypted in memory of the corresponding computing node and not persisted to non-volatile storage,
• determine, for each computing node that is executing the search query, whether data in the unencrypted files is responsive to the search query,
• compile responsive data to the search query to a result file and encrypt the result file, and
• deliver the result file to the participant data store that corresponds to the participant that submitted the search query.