Differentiable set to increase the memory capacity of recurrent neural net works

One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause:

• training, based on a set of sequential training data, a recurrent neural network that is equipped with a differentiable set data structure;
• wherein training the recurrent neural network comprises:
• performing one or both of:
• adding an element to the differentiable set data structure based, at least in part, on a hidden state of the recurrent neural network,
• performing a query over the differentiable set data structure based, at least in part, on the hidden state of the recurrent neural network;
• and
• after performing one or both of adding the element and performing the query, generating a prediction, based on output of the query, without using the hidden state of the recurrent neural network;
• wherein training the recurrent neural network produces a trained recurrent neural network;
• generating, by the recurrent neural network, a new query that contains a vector that represents a value in an unlabeled sequence that is syntactically valid, wherein the vector contains a plurality of weights that are less than 0.5 and one weight that is greater than 0.5;
• and
• based on the new query that contains the vector that represents the value, the trained recurrent neural network, and the differentiable set data structure, detecting that the unlabeled sequence is semantically invalid.