# US 2014/0156575 A1 · Method and Apparatus of Processing Data Using Deep Belief Networks Employing Low-Rank Matrix Factorization > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0156575 A1 - Document ID: US20140156575A1 - Status: Published application - Kind: A1 - Application number: 13/691,400 - Filed: 2012-11-30 - Published: 2014-06-05 - Earliest claimed: 2012-11-30 - Assignees: Nuance Communications, Inc. - Applicants: Nuance Communications, Inc. - Inventors: Tara N. Sainath; Ebru Arisoy; Bhuvana Ramabhadran - Web page: https://patentgrep.com/patent/US20140156575A1 ## CPC - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Deep belief networks are usually associated with a large number of parameters and high computational complexity. The large number of parameters results in a long and computationally consuming training phase. According to at least one example embodiment, low-rank matrix factorization is used to approximate at least a first set of parameters, associated with an output layer, with a second and a third set of parameters. The total number of parameters in the second and third sets of parameters is smaller than the number of sets of parameters in the first set. An architecture of a resulting artificial neural network, when employing low-rank matrix factorization, may be characterized with a low-rank layer, not employing activation function(s), and defined by a relatively small number of nodes and the second set of parameters. By using low rank matrix factorization, training is faster, leading to rapid deployment of the respective system. ## Claims ### Claim 1 A computer-implemented method of processing data, representing a real-world phenomenon, using an artificial neural network configured to model a real-world system or data pattern, the method comprising: applying a non-linear activation function to a weighted sum of input values at each node of at least one hidden layer of the artificial neural network; calculating a weighted sum of input values at each node of at least one low-rank layer of the artificial neural network without applying a non-linear activation function to the calculated weighted sum, the input values at each node of the at least one low-rank layer being output values from nodes of a last hidden layer of the at least one hidden layer; and generating output values by applying a non-linear activation function to a weighted sum of input values at each node of an output layer, the input values at each node of the output layer being output values from nodes of a last low-rank layer of the at least one low-rank layer of the artificial neural network. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the at least one low-rank layer and associated weighting coefficients are obtained by applying an approximation, using low rank matrix factorization, to weighting coefficients interconnecting the last hidden layer to the output layer in a baseline artificial neural network that does not include the at least one low-rank layer. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein the number of nodes of the at least one low-rank layer is fewer than the number of nodes of the last hidden layer. ### Claim 4 (depends on 1) The computer-implemented method of claim 1 further comprising: adjusting weighting coefficients associated with nodes of the at least one hidden layer, the at least one low-rank layer, and the output layer based at least in part on outputs of the artificial neural network and training data. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein adjusting weighting coefficients includes using a fine-tuning approach or a back-propagation approach. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the generated output values are indicative of probability values corresponding to a plurality of classes, the plurality of classes being represented by the nodes of the output layer. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the artificial neural network is a deep belief network. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the data includes speech data and the artificial neural network is used for speech recognition. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the data includes text data and the artificial neural network is used for language modeling. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the data includes image data and the artificial neural network is used for image processing. ### Claim 11 An apparatus for processing data, representing a real-world phenomenon, using an artificial neural network configured to model a real-world system or data pattern, the apparatus comprising: at least one processor; and at least one memory with computer code instructions stored thereon, the at least one processor and the at least one memory with the computer code instructions being configured to cause the apparatus to perform at least the following: apply a non-linear activation function to a weighted sum of input values at each node of at least one hidden layer of the artificial neural network; calculate a weighted sum of input values at each node of at least one low-rank layer of the artificial neural network without applying a non-linear activation function to the calculated weighted sum, the input values at each node of the at least one low-rank layer being output values from nodes of a last hidden layer of the at least one hidden layer; and generate output values by applying a non-linear activation function to a weighted sum of input values at each node of an output layer, the input values at each node of the output layer being output values from nodes of a last low-rank layer of the at least one low-rank layer of the artificial neural network. ### Claim 12 (depends on 11) The apparatus of claim 11, wherein the at least one low-rank layer and associated weighting coefficients are obtained by applying an approximation, using low rank matrix factorization, to weighting coefficients interconnecting the last hidden layer to the output layer in a baseline artificial neural network that does not include the at least one low-rank layer. ### Claim 13 (depends on 12) The apparatus of claim 12, wherein the number of nodes of the at least one low-rank layer is fewer than the number of nodes of the last hidden layer. ### Claim 14 (depends on 11) The apparatus of claim 11, wherein the at least one processor and the at least one memory, with the computer code instructions, being further configured to cause the apparatus to: adjust weighting coefficients associated with nodes of the at least one hidden layer, the at least one low-rank layer, and the output layer based at least in part on outputs of the artificial neural network and training data. ### Claim 15 (depends on 14) The apparatus of claim 14, wherein adjusting weighting coefficients includes using a fine-tuning approach or a back-propagation approach. ### Claim 16 (depends on 11) The apparatus of claim 11, wherein the generated output values are indicative of probability values corresponding to a plurality of classes, the plurality of classes being represented by the nodes of the output layer. ### Claim 17 (depends on 11) The apparatus of claim 11, wherein the artificial neural network is a deep belief network. ### Claim 18 (depends on 11) The apparatus of claim 11, wherein the data includes speech data and the artificial neural network is used for speech recognition. ### Claim 19 (depends on 11) The apparatus of claim 11, wherein the data includes text data and the artificial neural network is used for language modeling. ### Claim 20 A non-transitory computer-readable medium with computer code instructions stored thereon, the computer code instructions when executed by a processor, cause an apparatus to perform at least the following: applying a non-linear activation function to a weighted sum of input values at each node of at least one hidden layer of an artificial neural network; calculating a weighted sum of input values at each node of at least one low-rank layer of the artificial neural network without applying a non-linear activation function to the calculated weighted sum, the input values at each node of at least one low-rank layer being output values from nodes of a last hidden layer of the at least one hidden layer; and generating output values by applying a non-linear activation function to a weighted sum of input values at each node of an output layer, the input values at each node of the output layer being output values from nodes of a last low-rank layer among the at least one low-rank layer of the artificial neural network.