Generating a database system using distributed data sources to facilitate data retrieval

A computer-implemented method comprising:

• receiving, by a processing device, one or more messages from a plurality of data sources, each data source of the plurality of data sources providing the messages in a non-standardized format;
• converting, by the processing device, the one or more messages from the plurality of data sources into a standardized format by modifying a respective structure of at least a portion of the one or more messages to transform unstructured data into structured data;
• storing, by the processing device, the converted one or more messages in the standardized format as a dataset in a database system;
• providing, by the processing device, access by one or more subscribers to the database system to collect at least a portion of the dataset;
• outputting, by the processing device based on a request from at least one subscriber of the one or more subscribers, an interface that comprises at least the portion of the dataset to a user;
• determining, based on the request, whether contents of the database system are sufficient to resolve the request, the contents of the database system comprising the one or more messages and metadata associated with the one or more messages;
• and
• in response to determining that the contents of the database system are insufficient to resolve the request, communicating with an external data source to resolve the request by:
• identifying, based on requested information indicated in the request, a remaining portion of the requested information that is unavailable in the database system;
• determining that the external data source comprises the remaining portion of the requested information;
• and
• transmitting a retrieval request to the external data source to obtain the remaining portion of the requested information from the external data source such that the interface comprises the requested information.

A non-transitory computer-readable medium comprising program code executable by a processing device for causing the processing device to perform operations comprising:

• receiving one or more messages from a plurality of data sources, each data source of the plurality of data sources providing the messages in a non-standardized format;
• converting the one or more messages from the plurality of data sources into a standardized format by modifying a respective structure of at least a portion of the one or more messages to transform unstructured data into structured data;
• storing the converted one or more messages in the standardized format as a dataset in a database system;
• providing access by one or more subscribers to the database system to collect at least a portion of the dataset;
• outputting, based on a request from at least one subscriber of the one or more subscribers, an interface that comprises at least the portion of the dataset to a user;
• determining, based on the request, whether contents of the database system are sufficient to resolve the request, the contents of the database system comprising the one or more messages and metadata associated with the one or more messages;
• and
• in response to determining that the contents of the database system are insufficient to resolve the request, communicating with an external data source to resolve the request by:
• identifying, based on requested information indicated in the request, a remaining portion of the requested information that is unavailable in the database system;
• determining that the external data source comprises the remaining portion of the requested information;
• and
• transmitting a retrieval request to the external data source to obtain the remaining portion of the requested information from the external data source such that the interface comprises the requested information.