Time-Based Healthcare Data Management with Partitioned Storage and Compaction

A method of managing healthcare data in a cloud computing environment, the method comprising:

• receiving, at a staging platform, healthcare data from a plurality of healthcare data sources;
• partitioning the healthcare data into a plurality of data partitions based on at least one of:
• (a) one or more types of healthcare data indicated by metadata associated with healthcare data or (b) one or more sources of healthcare data indicated by metadata associated with healthcare data;
• and
• for a data partition of the plurality of data partitions:
• storing healthcare data of the partition in a plurality of time slices in a long-term storage data store, wherein a time slice of the plurality of time slices corresponds to a defined time period and contains healthcare data of the partition received during that defined time period;
• and
• compacting multiple time slices of the plurality of time slices into a larger historical time slice to reduce storage space in the long-term storage data store;
• and
• providing access to healthcare data stored in one or more time slices of the plurality of time slices and the larger historical time slice to one or more processing nodes that subscribe to healthcare data.

A system comprising:

• a set of one or more computing device;
• instructions for managing healthcare data in a cloud computing environment which, when executed, cause the set of one or more computing devices to perform a set of operations comprising:
• receiving, at a staging platform, healthcare data from a plurality of healthcare data sources;
• partitioning the healthcare data into a plurality of data partitions based on at least one of:
• (a) one or more types of healthcare data indicated by metadata associated with healthcare data or (b) one or more sources of healthcare data indicated by metadata associated with healthcare data;
• and
• for a data partition of the plurality of data partitions:
• storing healthcare data of the partition in a plurality of time slices in a long-term storage data store, wherein a time slice of the plurality of time slices corresponds to a defined time period and contains healthcare data of the partition received during that defined time period;
• and
• compacting multiple time slices of the plurality of time slices into a larger historical time slice to reduce storage space in the long-term storage data store;
• and
• providing access to healthcare data stored in one or more time slices of the plurality of time slices and the larger historical time slice to one or more processing nodes that subscribe to healthcare data
• wherein providing access to healthcare data is based on:
• generating routing rules defining one or more data types and one or more sources to which each processing node, of the one or more processing node, subscribes;
• identifying, based on the routing rules, a subset of healthcare data to be delivered with low latency to a first processing node;
• delivering the subset of healthcare data to a data sink associated with the first processing node;
• converting the subset of healthcare data into a local format specified by the data sink;
• and
• storing the converted subset of healthcare data in a processed storage engine accessible by a computing process associated with the first processing node.