High dimensional dense tensor representation for log data

A device, comprising:

• one or more memories;
• and
• one or more processors, coupled to the one or more memories, configured to:
• obtain a training corpus from a set of log data, wherein the training corpus includes alphanumeric characters;
• encode the training corpus to obtain a set of encoded data using vocabulary tokens that are based on unique characters included in the alphanumeric characters of the training corpus, wherein the set of encoded data is associated with a numeric format;
• calculate a sequence length based on a statistical parameter associated with the training corpus;
• generate a set of input sequences and a set of target sequences based on the set of encoded data, wherein each input sequence, from the set of input sequences, and each target sequence, from the set of target sequences, has a length equal to the sequence length;
• generate a training data set based on combining the set of input sequences and the set of target sequences, and shuffling data included in the combined set of input sequences and set of target sequences to obtain the training data set;
• train a deep neural network using the training data set and based on one or more hyperparameters to obtain a set of embedding tensors associated with an embedding layer of the deep neural network, wherein the deep neural network includes the embedding layer, a recurrent neural network layer, and a dense neural network layer;
• and
• perform an artificial intelligence operation using the set of embedding tensors to obtain information associated with the set of log data.

A non-transitory computer-readable medium storing a set of instructions, the set of instructions comprising:

• one or more instructions that, when executed by one or more processors of a device, cause the device to:
• detect one or more outlier data sets from pre-processed log data and remove the one or more outlier data sets from the pre-processed log data;
• obtain a training corpus based on concatenating the pre-processed log data, wherein the training corpus includes alphanumeric characters;
• encode the training corpus to obtain a set of encoded data using a set of vocabulary tokens that are based on unique characters included in the alphanumeric characters, wherein the encoded data is associated with a numeric format;
• calculate a sequence length based on a statistical parameter associated with the training corpus;
• generate a set of input sequences and a set of target sequences based on the set of encoded data, wherein each input sequence, from the set of input sequences, and each target sequence, from the set of target sequences, have a length equal to the sequence length;
• generate a training data set based on combining the set of input sequences and the set of target sequences into a tuple, partitioning the tuple into batches based on a batch size, and shuffling information included in the batches to obtain the training data set;
• and
• train a sequential deep neural network using the training data set and based on one or more hyperparameters to obtain a set of embedding tensors associated with an embedding layer of the sequential deep neural network, wherein the set of embedding tensors is a numerical representation of the pre-processed log data.