Computer-implemented method, computer program product and system for data analysis

A computer-implemented method for evaluating reliability of prediction given by a deep neural network for a new observation input to the deep neural network, the deep neural network comprising a plurality of hidden layers and being trained based on a training dataset including observations that are input to the deep neural network, the method comprising:

• receiving the new observation to be input to the deep neural network;
• obtaining a prediction given by the deep neural network for the received new observation;
• obtaining a second set of intermediate output values that are output from at least one of the plurality of hidden layers of the deep neural network by inputting the received new observation to the deep neural network;
• mapping, using a latent variable model stored in a storage medium, the second set of intermediate output values to a second set of projected values;
• determining whether the received new observation is an outlier with respect to the training dataset based on the latent variable model and the second set of projected values;
• and
• determining whether the prediction for the received new observation given by the deep neural network is a reliable prediction, wherein the prediction for the received new observation is determined to be unreliable if the received new observation is determined to be an outlier and the prediction for the received new observation is determined to be reliable if the received new observation is determined not to be an outlier;
• wherein the latent variable model stored in the storage medium is constructed by:
• obtaining first sets of intermediate output values that are output from said one of the plurality of hidden layers of the deep neural network, each of the first sets of intermediate output values obtained by inputting a different one of the observations included in at least a part of the training dataset to the deep neural network;
• and
• constructing the latent variable model using the first sets of intermediate output values, the latent variable model providing a mapping of the first sets of intermediate output values to first sets of projected values in a sub-space that has a dimension lower than a dimension of the first sets of the intermediate output values.

One or more non-transitory computer-readable media comprising computer-readable instructions that, when loaded and run on a computer, cause the computer to:

• obtain a deep neural network for processing data, the deep neural network trained based on a training dataset, and obtain at least a part of a training dataset used for training the deep neural network, the deep neural network comprising a plurality of hidden layers, the training dataset including observations that are input to the deep neural network;
• obtain first sets of intermediate output values that are output from at least one of the plurality of hidden layers of the deep neural network, each of the first sets of intermediate output values obtained by inputting a different one of the observations included in said part of the training dataset to the deep neural network;
• construct, via at least one processor, a latent variable model using the first sets of intermediate output values, the latent variable model providing a mapping of the first sets of intermediate output values to first sets of projected values in a sub-space that has a dimension lower than a dimension of the first sets of the intermediate output values;
• receive a new observation to be input to the deep neural network;
• obtain a prediction given by the deep neural network for the received new observation;
• obtain a second set of intermediate output values that are output from at least one of said plurality of hidden layers of the deep neural network by inputting the received new observation to the deep neural network;
• map, using the latent variable model, the second set of intermediate output values to a second set of projected values;
• determine, via the processor, whether the received new observation is an outlier with respect to the training dataset based on the latent variable model and the second set of projected values;
• and
• determining whether the prediction for the received new observation given by the deep neural network is a reliable prediction, wherein the prediction for the received new observation is determined to be unreliable if the received new observation is determined to be an outlier and the prediction for the received new observation is determined to be reliable if the received new observation is determined not to be an outlier.

A system for evaluating reliability of prediction given by a deep neural network for a new observation input to the deep neural network, the deep neural network comprising a plurality of hidden layers and being trained based on a training dataset including training observations that are input to the deep neural network, the system comprising:

• a storage medium storing a training dataset used for training a deep neural network;
• and
• a processor configured to:
• obtain the deep neural network trained using the training dataset;
• obtain at least a part of the training dataset stored in the storage medium;
• obtain first sets of intermediate output values that are output from at least one of the plurality of hidden layers of the deep neural network, each of the first sets of intermediate output values obtained by inputting a different one of the observations included in said part of the training dataset;
• construct a latent variable model using the first sets of intermediate output values, the latent variable model providing a mapping of the first sets of intermediate output values to first sets of projected values in a sub-space that has a dimension lower than a dimension of the first sets of the intermediate output values;
• receive the new observation to be input to the deep neural network;
• obtain a prediction given by the deep neural network for the received new observation;
• obtain a second set of intermediate output values that are output from at least one of said plurality of hidden layers of the deep neural network by inputting the received new observation to the deep neural network;
• map, using the latent variable model, the second set of intermediate output values to a second set of projected values;
• determine whether the received new observation is an outlier with respect to the training dataset based on the latent variable model and the second set of projected values and
• determine, whether the prediction for the received new observation given by the deep neural network is a reliable prediction based on determining, whether the received new observation is an outlier, wherein the prediction for the received new observation is determined to be unreliable if the received new observation is determined to be an outlier and determined reliable if the received new observation is determined not to be an outlier.