# US 12,182,719 B2 · Fixed, random, recurrent matrices for increased dimensionality in neural networks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,719 B2 - Document ID: US12182719B2 - Status: Granted - Kind: B2 - Application number: 16/940,857 - Filed: 2020-07-28 - Published: 2022-02-03 - Granted: 2024-12-31 - Earliest claimed: 2020-07-28 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Lorenz K. Muller; Pascal Stark; Stefan Abel - Web page: https://patentgrep.com/patent/US12182719B2 ## CPC - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/048: Activation functions - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/0499: Feedforward networks - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A method of operating a neural network. The input layer of the network may have n input nodes connected to output nodes via a hidden layer. The hidden layer may include m hidden nodes. The n input nodes may connect to a subset of k nodes of the m hidden nodes via respective synaptic connections, to which training weights are associated, which form an n×k input matrix Win, whereas a subset of m−k nodes of the hidden layer are not connected by any node of the input layer. Running the network may include performing a first matrix vector multiplication between the input matrix Win and a vector of values obtained in output of the input nodes and a second matrix vector multiplication between a fixed matrix Wrec of fixed weights and a vector of values obtained in output of the m nodes of the hidden layer. ## Claims ### Claim 1 A computer-implemented method of operating a neural network, the method comprising: training a neural network comprising successive layers, wherein: the successive layers include an input layer, a hidden layer, and an output layer; the input layer has n input nodes connected to output nodes of the output layer via the hidden layer; the hidden layer including m hidden nodes, wherein the neural network is trained through connecting the n input nodes to a subset of k nodes of the m hidden nodes via respective synaptic connections to reduce number of learned parameters during the training, wherein each node in the subset of k nodes is associated with a training weight, which is used to form an n×k input matrix Win, wherein the training weights for the m hidden nodes are changeable during the training, and wherein a predetermined subset of m−k nodes of the hidden layer are not connected by any node of the input layer, wherein each node in the m−k nodes is associated with a fixed weight, wherein the fixed weights for the m−k nodes remain unchanged during the training; and the neural network is set as a feed-forward neural network, the feed-forward neural network including a backpropagation learning algorithm, wherein the backpropagation learning algorithm comprises running the neural network so as to train the training weights of the n×k input matrix Win; and coupling input data into input nodes of the input layer. ### Claim 2 (depends on 1) The computer-implemented method according to claim 1, further comprising receiving a fixed matrix Wrec of the fixed weights. ### Claim 3 (depends on 1) The computer-implemented method according to claim 1, wherein output of the m hidden nodes is a vector. ### Claim 4 (depends on 1) The computer-implemented method according to claim 1, wherein wherein the subset of k nodes are connected by the n input nodes of the input layer. ### Claim 5 (depends on 4) The computer-implemented method according to claim 4, wherein output values y are obtained from input data x. ### Claim 6 (depends on 1) The computer-implemented method according to claim 1, wherein said successive layers comprise several hidden layers, including said hidden layer, each configured as said hidden layer, whereby the input nodes are connected to the output nodes via the several hidden layers. ### Claim 7 (depends on 1) The computer-implemented method according to claim 1, further comprising some of the fixed weights being random. ### Claim 8 (depends on 7) The computer-implemented method according to claim 7, wherein a complementary subset of the fixed weights is random. ### Claim 9 (depends on 7) The computer-implemented method according to claim 7, further comprising an echo state property. ### Claim 10 (depends on 9) The computer-implemented method according to claim 9, wherein said fixed weights comprise a spectral radius that is strictly less than 1. ### Claim 11 (depends on 1) The computer-implemented method according to claim 1, wherein the neural network is run so as to train the training weights of the n×k input matrix Win, in a supervised manner. ### Claim 12 (depends on 1) The computer-implemented method according to claim 1, wherein output values are inferred using a previous training. ### Claim 13 (depends on 1) The method according to claim 1, wherein said neural network is set as a multilayer perceptron or a convolutional neural network. ### Claim 14 (depends on 1) The method according to claim 1, wherein said neural network is set as an item-based autoencoder. ### Claim 15 A computer program product for operating a neural network, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor, so as to cause the processor to: train a neural network comprising successive layers, wherein: the successive layers include an input layer, a hidden layer, and an output layer; the input layer has n input nodes connected to output nodes of the output layer via the hidden layer; the hidden layer including m hidden nodes; and wherein the neural network is trained through connecting the n input nodes to a subset of k nodes of the m hidden nodes via respective synaptic connections to reduce number of learned parameters during the training, wherein each node in the subset of k nodes is associated with a training weight, which is used to form an n×k input matrix Win, wherein the training weights for the m hidden nodes are changeable during the training, and wherein a predetermined subset of m−k nodes of the hidden layer are not connected by any node of the input layer, wherein each node in the m−k nodes is associated with a fixed weight, wherein the fixed weights for the m−k nodes remain unchanged during the training; and the neural network is set as a feed-forward neural network, the feed-forward neural network including a backpropagation learning algorithm, wherein the backpropagation learning algorithm comprises running the neural network so as to train the training weights of the n×k input matrix Win; and couple input data into input nodes of the input layer. ### Claim 16 (depends on 15) The computer program product of claim 15, wherein input comprises a fixed matrix Wrec of the fixed weights. ### Claim 17 A neural network, comprising a processor operably connected to a memory, the memory containing program instructions to, when executed on the processor: train a neural network comprising successive layers, wherein: the successive layers include an input layer, a hidden layer, and an output layer; the input layer has n input nodes connected to output nodes of the output layer via the hidden layer; the hidden layer including m hidden nodes; and wherein the neural network is trained through connecting the n input nodes to a subset of k nodes of the m hidden nodes via respective synaptic connections to reduce number of learned parameters during the training, wherein each node in the subset of k nodes is associated with a training weight, which is used to form an n×k input matrix Win, wherein the training weights for the m hidden nodes are changeable during the training, and wherein a predetermined subset of m−k nodes of the hidden layer are not connected by any node of the input layer, wherein each node in the m−k nodes is associated with a fixed weight, wherein the fixed weights for the m−k nodes remain unchanged during the training; and the neural network is set as a feed-forward neural network, the feed-forward neural network including a backpropagation learning algorithm, wherein the backpropagation learning algorithm comprises running the neural network so as to train the training weights of the n×k input matrix Win; and couple input data into input nodes of the input layer. ### Claim 18 (depends on 17) The neural network of claim 17, wherein the input comprises a fixed matrix Wrec of the fixed weights. ### Claim 19 (depends on 1) The computer-implemented method according to claim 1, wherein the backpropagation learning algorithm comprises only modifies the weights of the n×k input matrix Win. ### Claim 20 (depends on 1) The computer-implemented method according to claim 1, wherein the backpropagation learning algorithm comprises does not modify the subset of m−k nodes of the hidden layer that are not connected by any node of the input layer.