Aggregation, partitioning, and management of healthcare data for efficient storage and processing

A computer-implemented system for aggregating, archiving, and managing healthcare data, the computer-implemented system comprising:

• one or more disparate originating healthcare data sources;
• one or more processing nodes;
• a staging component that receives and indexes one or more pieces of healthcare data from a plurality of disparate originating healthcare data sources so that it is optimized for low-latency processing by storing the data in a raw form with additional metadata describing a source and a time received for the data;
• an extraction component that extracts one or more pieces of healthcare data received and indexed at the staging component from a plurality of disparate originating healthcare data sources;
• an aggregating component that aggregates a subset of the healthcare data extracted by the extraction component, the aggregating component aggregating the subset of healthcare data having the same type into batches;
• a partitioning component that sorts the batches by originating healthcare data source;
• and
• a long-term storage data store for storing the batches;
• wherein the staging component further receives a request from the one or more processing nodes for at least one of the batches;
• wherein the staging component further accesses the at least one of the batches in the long-term storage data store;
• and
• wherein the staging component further delivers the at least one of the batches of healthcare data to the processing node where the at least one of the requested batches will be processed to produce a clinically relevant outcome.

One or more non-transitory computer-storage media having computer-executable instructions embodied thereon that, when executed, perform a method of aggregating, partitioning, and storing healthcare data, the method comprising:

• extracting a set of healthcare data from one or more healthcare data sources;
• filtering and removing duplicate copies of one or more pieces of the set of healthcare data;
• aggregating a subset of the set of healthcare data having the same type;
• partitioning the aggregated subset of healthcare data by healthcare data source;
• and
• storing the aggregated and partitioned subset of healthcare data in one or more long-term storage data stores;
• receiving a request from a processing node for a batch of healthcare data of a particular type and healthcare data source;
• and
• accessing the batch of healthcare data in the long-term storage data store;
• and
• delivering the batch of healthcare data to the processing node where the requested batch of healthcare data will be processed to produce a clinically relevant outcome.