Fast ingest, archive and retrieval systems, method and computer programs

A data processing method comprising:

• ingesting data for archiving;
• partitioning by a pre-archive filter the ingested data using predetermined queryable fields;
• storing the partitioned data in a short term storage device only until archiving of the data from the short term storage device to a long term storage is completed;
• processing the ingested data to create metadata relating to the ingested data;
• storing the metadata in a metadata catalog of a database;
• processing a query for querying against data stored in the short term storage device and the long term storage in the same way such that the data is always in a queryable state through the rest of the data's lifecycle from the partitioning;
• processing the partitioned data to store it in the long term storage;
• updating the metadata catalog to reference the archived data;
• and
• correlating by an inference engine a first data and a second data from the data to generate a first relationship pair, correlating the second data and a third data from the data to generate a second relationship pair, and correlating the first relationship pair and the second relationship pair to generate a third relationship pair correlating the first data and the third data, the inference engine operating in parallel with the pre-archive filter,
• wherein in the correlating, n degrees of correlation are used to correlate multiple dimensions of one-to-one or one-to-many inference metadata to produce inference relationships,
• wherein the data is queryable during the correlating, and
• wherein queryable includes user based searching using keywords.

A non-transitory computer-readable medium comprising:

• a first code segment for ingesting data for archiving;
• a second code segment for partitioning the ingested data using predetermined queryable fields;
• a third code segment for storing the partitioned data in a short term storage device only until archiving of the data from the short term storage device to a long term storage is completed;
• a fourth code segment for processing the ingested data to create metadata relating to the ingested data;
• a fifth code segment for processing a query for querying against data stored in the short term storage device and the long term storage in the same way such that the data is always in a queryable state through the rest of the data's lifecycle from the partitioning;
• a sixth code segment for storing data archived from the short term storage device to the long term storage;
• a seventh code segment for updating a metadata catalog to reference the archived data;
• and
• an eighth code segment for correlating a first data and a second data from the data to generate a first relationship pair, correlating the second data and a third data from the data to generate a second relationship pair, and correlating the first relationship pair and the second relationship pair to generate a third relationship pair correlating the first data and the third data, the eighth code segment operating in parallel with the second code segment,
• wherein in the correlating, n degrees of correlation are used to correlate multiple dimensions of one-to-one or one-to-many inference metadata to produce inference relationships,
• wherein the data is queryable during the correlating, and
• wherein queryable includes user based searching using keywords.

A non-transitory computer-readable medium for generating code, comprising:

• a first code segment for processing data derived from an interface control document that defines formats for data types and code foundation data to automatically generate an extensible markup language (XML) file and generate source code having differing functionality that shares consistent data structures and methods using the XML file;
• a second code segment for partitioning the processed data using predetermined queryable fields;
• a third code segment for storing partitioned data in a short term storage device only until archiving of the data from the short term storage device to a long term storage is completed;
• a fourth code segment for processing a query for querying against data stored in the short term storage device and the long term storage in the same way such that the data is always in a queryable state through the rest of the data's lifecycle from the partitioning;
• a fifth code segment for storing data archived from the short term storage device to the long term storage;
• and
• a sixth code segment for correlating a first data and a second data from the data to generate a first relationship pair, correlating the second data and a third data from the data to generate a second relationship pair, and correlating the first relationship pair and the second relationship pair to generate a third relationship pair correlating the first data and the third data, the sixth code segment operating in parallel with the second code segment,
• wherein in the correlating, n degrees of correlation are used to correlate multiple dimensions of one-to-one or one-to-many inference metadata to produce inference relationships,
• wherein the data is queryable during the correlating, and
• wherein queryable includes user based searching using keywords.