Managing data ingestion

A computer program product for use at a computer system, the computer program product for implementing a method for supplementing a data consumer defined data entity with additional data from a new data source, the data consumer defined data entity spanning one or more ingested data sets from one or more data sources, the computer program product comprising one or more computer storage media having stored thereon computer-executable instructions that, when executed at a processor, cause the computer system to perform the method, including the following:

• receive a new data set from the new data source, the new data set in a raw data format used by the new data source, the new data source in addition to the one or more data sources;
• responsive to receiving the new data set from the new data source:
• ingest the new data set in the raw data format, ingesting the new data set including utilizing a combined access mechanism and security context matched to the new data source;
• convert the new data set into a common format using a formatting plug-in configured to understand the raw data formate;
• and
• store new data set into storage, the storage including the one or more ingested data sets, the one or more ingested data sets having been previously formatted into the common format from one or more other raw data formats used by the one or more data sources, the one or more data sets previously ingested from the one or more data sources using combined access mechanisms and security contexts matched to each of the one or more data sources;
• and
• apply a schema to stored data in the common format, the stored data from the new ingested data set and the one or more ingested data sets, the stored data associated with data consumer selected attributes included in the data consumer defined data entity.

A computer system, the computer system comprising:

• one or more processors;
• system memory coupled to the one or more hardware processors, the system memory storing instructions that are executable by the one or more hardware processors;
• the one or more hardware processors executing the instructions stored in the system memory to supplementing a data consumer defined data entity with additional data from a new data source, the data consumer defined data entity spanning one or more ingested data sets from one or more data sources, including the following:
• receive a new data set from the new data source, the new data set in a raw data format used by the new data source, the new data source in addition to the one or more data sources;
• responsive to receiving the new data set from the new data source:
• ingest the new data set in the raw data format, ingesting the new-data set including utilizing a combined access mechanism and security context matched to the new data source;
• convert the new data set into a common format using a formatting plug-in configured to understand the raw data format;
• and
• store new data set into storage, the storage including the one or more ingested data sets, the one or more ingested data sets having been previously formatted into the common format from one or more other raw data formats used by the one or more data sources, the one or more data sets previously ingested from the one or more data sources using combined access mechanisms and security contexts matched to each of the one or more data sources;
• and
• apply a schema to stored data in the common format, the stored data from the new ingested data set and the one or more ingested data sets, the stored data associated with data consumer selected attributes included in the data consumer defined data entity.