Automated data ingestion using an autoencoder

A non-transitory computer-readable storage medium storing instructions that when executed by a processor cause the processor to:

• generate, by a computing model, a transformed dataset based on a first dataset comprising numeric values, the computing model to convert the numeric values from a first data type to a second data type;
• generate a training dataset comprising the first dataset as an input dataset and the transformed dataset as an output dataset;
• train an autoencoder comprising a latent vector to transform the input dataset from the first data type to the second data type;
• determine, by a statistical model, an accuracy of the trained autoencoder based on an output of the trained autoencoder and the input dataset;
• and
• determine that the accuracy of the trained autoencoder exceeds a threshold accuracy;
• receive, by the trained autoencoder during a first time interval, streaming data comprising numeric values from a processing pipeline;
• determine, by the trained autoencoder during the first time interval, a maximum value and a minimum value of a first subset of the numeric values of the streaming data;
• and
• normalize, by the trained autoencoder during the first time interval, a second subset of the numeric values of the streaming data to be within the determined maximum and minimum values, wherein the trained autoencoder normalizes the second subset of the numeric values during the first time interval.

A method, comprising:

• generating, by a computing model executing on a computer processor, a transformed dataset based on a first dataset comprising numeric values, the computing model to convert the numeric values from a first data type to a second data type;
• generating a training dataset comprising the first dataset as an input dataset and the transformed dataset as an output dataset;
• training an autoencoder comprising a latent vector to transform the input dataset from the first data type to the second data type;
• determining, by a statistical model based on an output of the trained autoencoder and the input dataset, an accuracy of the trained autoencoder;
• determining that the accuracy of the trained autoencoder exceeds a threshold accuracy;
• receiving, by the trained autoencoder during a first time interval, streaming data comprising numeric values from a processing pipeline;
• determining, by the trained autoencoder during the first time interval, a maximum value and a minimum value of a first subset of the numeric values of the streaming data;
• and
• normalizing, by the trained autoencoder during the first time interval, a second subset of the numeric values of the streaming data to be within the determined maximum and minimum values, wherein the trained autoencoder normalizes the second subset of the numeric values during the first time interval.