Automated dynamic data extraction, distillation, and enhancement

A method of dynamic data extraction, distillation, and enhancement, the method comprising:

• receiving, by an allocator stored in a non-transitory memory and executable by a processor, a plurality of files in a plurality of formats from one or more data sources;
• for a first file of the plurality of files, determining, by the allocator, a native format of the first file;
• assigning, by the allocator, the first file to a first stream of a plurality of streams based on the native format of the first file, wherein each of the plurality of streams corresponds to one native format;
• for the first file of the plurality of files, identifying and extracting, by one or more extractors stored in a non-transitory memory and executable by a processor, at least one embedded object of a different format than the native format of the first file;
• assigning, by the allocator, the at least one embedded object of the different format to a second stream of the plurality of streams based on the different format of the at least one embedded object, wherein the first and second streams are different;
• processing, by a first stream processor, data from the first stream;
• adding, by the first stream processor, first attributes to the data from the first stream to create first enhanced data, wherein the first attributes comprise characteristics of the data being processed from the first stream;
• transmitting, by the first stream processor, the first enhanced data back into the first stream;
• processing, by a second stream processor, data including the at least one embedded object from the second stream;
• adding, by the second stream processor, second attributes to the data including the at least one embedded object from the second stream to create second enhanced data, wherein the second attributes comprise characteristics of the data being processed from the second stream, and wherein the first attributes and the second attributes comprise machine-learning derived characteristics that create relationships between data within and across the plurality of streams;
• transmitting, by the second stream processor, the second enhanced data back into the second stream;
• receiving, by a query component, a query directed to at least the first stream and the second stream after the first enhanced data is transmitted back into the first stream and the second enhanced data is transmitted back into the second stream;
• processing, by the query component, the query by applying one or more filters to data including the first enhanced data in the first stream and the second enhanced data in the second stream;
• and
• returning, by the query component, a response to the query based on the processing of the query, wherein the response includes results based on the first and second attributes added by the first and second stream processors.

A method of dynamic data extraction, distillation, and enhancement, the method comprising:

• receiving, by an allocator stored in a non-transitory memory and executable by a processor, a plurality of files in a plurality of formats from one or more data sources;
• for a first file of the plurality of files, determining, by the allocator, a native format of the first file;
• assigning, by the allocator, the first file to a first stream of a plurality of streams based on the native format of the first file, wherein each of the plurality of streams corresponds to one native format;
• for the first file in the first stream, identifying, by one or more extractors stored in a non-transitory memory and executable by a processor, at least one embedded link to a first external data;
• accessing and retrieving, by the one or more extractors, the external data based on the at least one embedded link;
• assigning, by the allocator, the external data to a second stream of the plurality of streams based on a format of the external data, wherein the first stream and the second stream are different;
• processing, by a first stream processor of a plurality of stream processors, data from the first stream;
• adding, by the first stream processor, first attributes to the data from the first stream to create first enhanced data, wherein the first attributes comprise characteristics of the data being processed from the first stream;
• transmitting, by the first stream processor, the first enhanced data back into the first stream;
• processing, by a second of the plurality of stream processors, data including the external data from the second stream;
• adding, by the second stream processor, second attributes to the data including the external data from the second stream to create second enhanced data, wherein the second attributes comprise characteristics of the external data being processed from the first stream, and wherein the first attributes and the second attributes comprise machine-learning derived characteristics that create relationships between data within and across the plurality of streams;
• transmitting, by the second stream processor, the second enhanced data back into the second stream;
• receiving, by a query component, a query directed to at least the first stream and the second stream after the first enhanced data is transmitted back into the first stream and the second enhanced data is transmitted back into the second stream;
• processing, by the query component, the query by applying one or more filters to data including the first enhanced data in the first stream and the second enhanced data in the second stream;
• and
• returning, by the query component, a response to the query based on the processing of the query, wherein the response includes results based on the first and second attributes added by the first and second stream processors.