# US 2020/0401888 A1 · Time series prediction with confidence estimates using sparse recurrent mixture density networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0401888 A1
- Document ID: US20200401888A1
- Status: Published application
- Kind: A1
- Application number: 16/909,262
- Filed: 2020-06-23
- Published: 2020-12-24
- Earliest claimed: 2019-06-24
- Assignees: Tata Consultancy Services Limited
- Applicants: Tata Consultancy Services Limited
- Inventors: Narendhar GUGULOTHU; Easwara Naga SUBRAMANIAN; Sanjay Purushottam BHAT
- Web page: https://patentgrep.com/patent/US20200401888A1

## CPC

- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

This disclosure relates generally to system and method for time series prediction using a sparse recurrent mixture density network (RMDN), such as sparse LSTM-MDN and a sparse ED-MDN, for accurate forecasting of a high variability time series. The disclosed sparse RMDN has the ability to handle high-dimensional input features, capture trend shifts and high variability present in the data, and provide a confidence estimate of the forecast. A high-dimensional time series data is passed through a feedforward layer, which performs dimensionality reduction in an unsupervised manner by inducing sparsity on weights of the feedforward layer. The resultant low-dimensional time series is fed through recurrent layers to capture temporal patterns. These recurrent layers also aid in learning latent representation of the input data. Thereafter, a mixture density network (MDN) is used to model the variability and trend shifts present in the input and it also estimates the confidence of the predictions.

## Claims

### Claim 1

A processor implemented method for time-series prediction using a sparse recurrent mixture density networks (RMDN) model, the method comprising:
iteratively predicting, via one or more hardware processors, time series in a plurality of iterations using a data set comprising a plurality of high-dimensional time series, the plurality of high-dimensional time series comprising a first set of high-dimensional time series associated with a training data and a second set of the high-dimensional time series associated with a validation data, each iteration of the plurality of iterations comprising:
passing, through a feedforward layer of the sparse RMDN model, a high-dimensional time series from amongst the plurality of high-dimensional time series, the sparse RMDN model comprising the feedforward layer, a recurrent neural network (RNN) and a mixture density network (MDN), the feedforward layer comprising a plurality of units associated with a plurality of distinct weights learnt by training the sparse RMDN model, the sparse RMDN model being trained by imposing Lasso penalty on the plurality of weights of the feedforward layer to determine a set of features associated with the time series in an unsupervised manner;
performing, by the feedforward layer, dimensionality reduction of the high-dimensional time series to obtain a reduced dimensional time series, the feedforward layer comprising a number of the plurality of units equal to a fraction of the number of features in the set of features to perform the dimensionality reduction;
feeding, through the RNN, the reduced dimensional time series to obtain latent representation of the high-dimensional time-series, the latent representation captures temporal patterns from the reduced dimensional time series;
feeding the latent representation of the high-dimensional time series to a mixture of Gaussians comprising a plurality of Gaussian components to predict a plurality of parameters associated with the plurality of Gaussian components in the mixture, the plurality of parameters comprising a plurality of probability values, a plurality of mean values and a plurality of standard deviation values associated with the plurality of the Gaussian components;
selecting a Gaussian component from amongst the plurality of Gaussian components that is associated with a highest value of probability from amongst the plurality of probability values, wherein the mean of the selected Gaussian component is selected as prediction of the time-series and the standard deviation of the Gaussian component is selected for confidence estimation of the prediction for the iteration;
computing a value of a loss function indicative of error in the prediction of the time-series using the plurality of parameters, the loss function being one of a training loss function and a validation loss function; and
updating, via the one or more hardware processors, a plurality of weights of the sparse RMDN model using the value of the training loss function after each iteration of the plurality of iterations associated with the training data for prediction of the time-series.

### Claim 2 (depends on 1)

The processor implemented method of claim 1, further comprising validating the prediction by the RMDN model, wherein validating comprises:
iteratively predicting the time series in a second plurality of iterations from amongst the plurality of iterations using the validation data set;
computing, based on the predicted time series, the validation loss function indicative of error in validation; and
selecting an iteration from amongst the second plurality of iterations for time series prediction based on the validation loss function value.

### Claim 3 (depends on 1)

The processor implemented method of claim 1, wherein computing the value of the loss function comprises:
determining a conditional probability distribution for a plurality of time steps in a forecast time horizon associated with the time-series based on the plurality of parameters;
minimizing a negative log likelihood function of the conditional probability distribution; and
computing the loss function along with the Lasso penalty on the plurality of weights of the feedforward layer based on the negative likelihood function, wherein, the negative log likelihood function of the conditional probability distribution is represented by the equation:
L
RMDN
=
-
1
N

∑
i
=
1
N


log


P

(
y
t
+
1
,
…
,
t
+
p
i
|
x
1
,
…
,
t
i
;
z
t
i
)
,
and where the loss function comprises:
L
=
L
RMDN
+
λ
d
×
r


Wf

1
where, superscript i denotes the ith sample, and N is the total number of samples in a training set, and λ is a regularization parameter that controls the level of sparsity in Wf.

### Claim 4 (depends on 1)

The processor implemented method of claim 1, wherein the set of parameters associated with the Gaussian component are estimated based on the equations:
ρt′,K(zt)=softmax(Wρ·zt+bρ),
μt′,K(zt)=Wμ·zt+bμ,
σt′,K(zt)=exp(Wσ·zt+bσ)
where t∈[t+1, . . . , t+p], and Wσ, Wμ, Wρ, are the learned parameters of the MDN with μ⋅,k and σ⋅,k, representing mean and standard deviation of the kth Gaussian component, respectively.

### Claim 5 (depends on 4)

The processor implemented method of claim 4, wherein the conditional probability distribution is represented as:
P(yt+1, . . . t+p|x1, . . . |t);zt)=Πt′=t+1t+pΣk=1Kρt′,k(zt)Nt(yt′;μt′,k(zt),σt′,k(zt))
where yt+1 . . . ; t+p are values of the time series to be predicted

### Claim 6 (depends on 1)

The processor implemented method of claim 1, wherein the Lasso penalty comprises imposing sparsity on the plurality of weights of the feedforward layer by restricting a fraction of the weights to be close to zero to result in unsupervised feature selection.

### Claim 7 (depends on 1)

The processor implemented method of claim 1, wherein the sparse RMDN model comprises one of a sparse LSTM model and a sparse ED model,
wherein the sparse LSTM model comprises the feedforward layer with Lasso sparsity constraints on the plurality of distinct weights and a LSTM as the RNN, and
wherein the sparse ED comprises the feedforward layer with LASSO sparsity constraints on the plurality of distinct weights and an encoder decoder as the RNN.

### Claim 8

A system for time-series prediction using a sparse recurrent mixture density networks (RMDN) model, comprising:
one or more memories; and
one or more first hardware processors, the one or more first memories coupled to the one or more first hardware processors, wherein the one or more first hardware processors are configured to execute programmed instructions stored in the one or more first memories to:
iteratively predict time series in a plurality of iterations using a data set comprising a plurality of high-dimensional time series, the plurality of high-dimensional time series comprising a first set of high-dimensional time series associated with a training data and a second set of the high-dimensional time series associated with a validation data, wherein in each iteration of the plurality of iterations, the one or more hardware processors are configured by the instructions to:
pass, through a feedforward layer of the sparse RMDN model, a high-dimensional time series from amongst the plurality of high-dimensional time series, the sparse RMDN model comprising the feedforward layer, a recurrent neural network (RNN) and a mixture density network (MDN), the feedforward layer comprising a plurality of units associated with a plurality of distinct weights learnt by training the sparse RMDN model, the sparse RMDN model being trained by imposing Lasso penalty on the plurality of weights of the feedforward layer to determine a set of features associated with the time series in an unsupervised manner;
perform, by the feedforward layer, dimensionality reduction of the high-dimensional time series to obtain a reduced dimensional time series, the feedforward layer comprising a number of the plurality of units equal to a fraction of the number of features in the set of features to perform the dimensionality reduction;
feed, through the RNN, the reduced dimensional time series to obtain latent representation of the high-dimensional time-series, the latent representation captures temporal patterns from the reduced dimensional time series;
feed the latent representation of the high-dimensional time series to a mixture of Gaussians comprising a plurality of Gaussian components to predict a plurality of parameters associated with the plurality of Gaussian components in the mixture, the plurality of parameters comprising a plurality of probability values, a plurality of mean values and a plurality of standard deviation values associated with the plurality of the Gaussian components;
select a Gaussian component from amongst the plurality of Gaussian components that is associated with a highest value of probability from amongst the plurality of probability values, wherein the mean of the selected Gaussian component is selected as prediction of the time-series and the standard deviation of the Gaussian component is selected for confidence estimation of the prediction for the iteration;
compute a value of a loss function indicative of error in the prediction of the time-series using the plurality of parameters, the loss function being one of a training loss function and a validation loss function; and
update a plurality of weights of the sparse RMDN model using the value of the training loss function after each iteration of the plurality of iterations associated with the training data for prediction of the time-series.

### Claim 9 (depends on 8)

The system of claim 8, wherein the one or more hardware processors are further configured by the instructions to validate the prediction by the RMDN model, and wherein to validate, the one or more hardware processors are further configured by the instructions to:
iteratively predict the time series in a second plurality of iterations from amongst the plurality of iterations using the validation data set;
compute, based on the predicted time series, the validation loss function indicative of error in validation; and
select an iteration from amongst the second plurality of iterations for time series prediction based on the validation loss function value.

### Claim 10 (depends on 8)

The system of claim 8, wherein to compute the value of the loss function, wherein the one or more hardware processors are further configured by the instructions to:
determine a conditional probability distribution for a plurality of time steps in a forecast time horizon associated with the time-series based on the plurality of parameters;
minimize a negative log likelihood function of the conditional probability distribution; and
compute the loss function along with the Lasso penalty on the plurality of weights of the feedforward layer based on the negative likelihood function, wherein, the negative log likelihood function of the conditional probability distribution is represented by the equation:
L
RMDN
=
-
1
N

∑
i
=
1
N


log


P

(
y
t
+
1
,
…
,
t
+
p
i
|
x
1
,
…
,
t
i
;
z
t
i
)
,
and where the loss function comprises:
L
=
L
RMDN
+
λ
d
×
r


Wf

1
where, superscript i denotes the ith sample, and N is the total number of samples in a training set, and λ is a regularization parameter that controls the level of sparsity in Wf.

### Claim 11 (depends on 8)

The system of claim 8, wherein the set of parameters associated with the Gaussian component are estimated based on the equations:
ρt′,K(zt)=softmax(Wρ·zt+bρ),
μt′,K(zt)=Wμ·zt+bμ,
σt′,K(zt)=exp(Wσ·zt+bσ)
where t∈[t+1, . . . , t+p], and Wσ, Wμ, Wρ, are the learned parameters of the MDN with μ⋅,k and σ⋅,k, representing mean and standard deviation of the kth Gaussian component, respectively.

### Claim 12 (depends on 11)

The system of claim 11, wherein the conditional probability distribution is represented as:
P(yt+1, . . . t+p|x1, . . . |t);zt)=Πt′=t+1t+pΣk=1Kρt′,k(zt)Nt(yt′;μt′,k(zt),σt′,k(zt))
where yt+1 . . . ; t+p are values of the time series to be predicted

### Claim 13 (depends on 8)

The system of claim 8, wherein the Lasso penalty comprises imposing sparsity on the plurality of weights of the feedforward layer by restricting a fraction of the weights to be close to zero to result in unsupervised feature selection.

### Claim 14 (depends on 8)

The system of claim 8, wherein the sparse RMDN model comprises one of a sparse LSTM model and a sparse ED model,
wherein the sparse LSTM model comprises the feedforward layer with Lasso sparsity constraints on the plurality of distinct weights and a LSTM as the RNN, and
wherein the sparse ED comprises the feedforward layer with Lasso sparsity constraints on the plurality of distinct weights and an encoder decoder as the RNN.

### Claim 15

One or more non-transitory machine readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:
iteratively predicting, via one or more hardware processors, time series in a plurality of iterations using a data set comprising a plurality of high-dimensional time series, the plurality of high-dimensional time series comprising a first set of high-dimensional time series associated with a training data and a second set of the high-dimensional time series associated with a validation data, each iteration of the plurality of iterations comprising:
passing, through a feedforward layer of the sparse RMDN model, a high-dimensional time series from amongst the plurality of high-dimensional time series, the sparse RMDN model comprising the feedforward layer, a recurrent neural network (RNN) and a mixture density network (MDN), the feedforward layer comprising a plurality of units associated with a plurality of distinct weights learnt by training the sparse RMDN model, the sparse RMDN model being trained by imposing Lasso penalty on the plurality of weights of the feedforward layer to determine a set of features associated with the time series in an unsupervised manner;
performing, by the feedforward layer, dimensionality reduction of the high-dimensional time series to obtain a reduced dimensional time series, the feedforward layer comprising a number of the plurality of units equal to a fraction of the number of features in the set of features to perform the dimensionality reduction;
feeding, through the RNN, the reduced dimensional time series to obtain latent representation of the high-dimensional time-series, the latent representation captures temporal patterns from the reduced dimensional time series;
feeding the latent representation of the high-dimensional time series to a mixture of Gaussians comprising a plurality of Gaussian components to predict a plurality of parameters associated with the plurality of Gaussian components in the mixture, the plurality of parameters comprising a plurality of probability values, a plurality of mean values and a plurality of standard deviation values associated with the plurality of the Gaussian components;
selecting a Gaussian component from amongst the plurality of Gaussian components that is associated with a highest value of probability from amongst the plurality of probability values, wherein the mean of the selected Gaussian component is selected as prediction of the time-series and the standard deviation of the Gaussian component is selected for confidence estimation of the prediction for the iteration;
computing a value of a loss function indicative of error in the prediction of the time-series using the plurality of parameters, the loss function being one of a training loss function and a validation loss function; and
updating, via the one or more hardware processors, a plurality of weights of the sparse RMDN model using the value of the training loss function after each iteration of the plurality of iterations associated with the training data for prediction of the time-series.