# US 9,519,858 B2 · Feature-augmented neural networks and applications of same > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,519,858 B2 - Document ID: US9519858B2 - Status: Granted - Kind: B2 - Application number: 13/763,701 - Filed: 2013-02-10 - Published: 2014-08-14 - Granted: 2016-12-13 - Earliest claimed: 2013-02-10 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Corporation - Inventors: Geoffrey G. Zweig; Tomas Mikolov - Web page: https://patentgrep.com/patent/US9519858B2 ## CPC - G06N 3/08: Learning methods - G06F 40/44: Statistical methods, e.g. probability models - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/0499: Feedforward networks - G06N 3/09: Supervised learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A system is described herein which uses a neural network having an input layer that accepts an input vector and a feature vector. The input vector represents at least part of input information, such as, but not limited to, a word or phrase in a sequence of input words. The feature vector provides supplemental information pertaining to the input information. The neural network produces an output vector based on the input vector and the feature vector. In one implementation, the neural network is a recurrent neural network. Also described herein are various applications of the system, including a machine translation application. ## Claims ### Claim 1 A method performed using one or more processing devices, the method comprising: receiving a word input vector at an input layer of a neural network, the word input vector representing an individual word from an input sequence of words; receiving a topic feature vector at the input layer of the neural network, the topic feature vector being separate from the word input vector and representing topics expressed in the input sequence of words; using the neural network to generate an output vector at an output layer of the neural network based at least on the word input vector and the topic feature vector, wherein using the neural network includes, by a hidden layer of the neural network: modifying the word input vector using a first learned matrix; and modifying the topic feature vector using a second learned matrix that is separate from the first learned matrix, wherein the output vector represents a word probability given the word input vector and the topic feature vector; and performing a natural language processing operation based at least on the word probability represented by the output vector. ### Claim 2 (depends on 1) The method of claim 1, wherein using the neural network includes: by the hidden layer of the neural network, modifying a time-delayed hidden-state vector with a third learned matrix, wherein the time-delayed hidden-state vector represents an output of the hidden layer in a prior time instance, wherein the word input vector, the topic feature vector, and the time-delayed hidden-state vector are separate vectors, and wherein the first learned matrix, the second learned matrix, and the third learned matrix are separate matrices. ### Claim 3 (depends on 2) The method of claim 2, wherein using the neural network includes, by the output layer of the neural network: modifying the output of the hidden layer with a fourth learned matrix; and modifying the topic feature vector with a fifth learned matrix, wherein the first learned matrix, the second learned matrix, the third learned matrix, the fourth learned matrix, and the fifth learned matrix are separate matrices. ### Claim 4 (depends on 2) The method of claim 2, wherein using the neural network includes, by the hidden layer: performing a first multiplication operation of the word input vector by the first learned matrix to generate a first multiplication output; performing a second multiplication operation of the topic feature vector by the second learned matrix to generate a second multiplication output; performing a third multiplication operation of the time-delayed hidden-state vector by the third learned matrix to generate a third multiplication output; and summing the first multiplication output, the second multiplication output, and the third multiplication output to generate the output of the hidden layer. ### Claim 5 (depends on 1) The method of claim 1, further comprising: generating the topic feature vector using a Latent Dirichlet Allocation (LDA) technique; and as subsequent words from the input sequence are processed using the neural network, incrementally generating next topic feature vectors based at least on previous feature vectors. ### Claim 6 (depends on 5) The method of claim 5, wherein said incrementally generating comprises applying a decay factor to previous topic feature vectors for previous words that have already been processed by the neural network. ### Claim 7 (depends on 1) The method of claim 1, wherein the input sequence of words is part of an input document. ### Claim 8 A system comprising: at least one processing device; and at least one computer readable medium storing instructions which, when executed by the at least one processing device, cause the at least one processing device to: receive a word input vector at an input layer of a neural network, the word input vector representing an individual word from an input sequence of words; receive a topic feature vector at the input layer of the neural network, the topic feature vector being separate from the word input vector and representing topics expressed in the input sequence of words; use the neural network to generate an output vector at an output layer of the neural network based at least on the word input vector and the topic feature vector, wherein using the neural network includes, by a hidden layer of the neural network: modifying the word input vector using a first learned matrix; and modifying the topic feature vector using a second learned matrix that is separate from the first learned matrix, wherein the output vector represents a word probability given the word input vector and the topic feature vector; and perform a natural language processing operation based at least on the word probability represented by the output vector. ### Claim 9 (depends on 8) The system of claim 8, wherein the instructions, when executed by the at least one processing device, cause the at least one processing device to: by the hidden layer of the neural network, modify a time-delayed hidden-state vector with a third learned matrix, wherein the time-delayed hidden-state vector represents an output of the hidden layer in a prior time instance, wherein the word input vector, the topic feature vector, and the time-delayed hidden-state vector are separate vectors, and wherein the first learned matrix, the second learned matrix, and the third learned matrix are separate matrices. ### Claim 10 (depends on 9) The system of claim 9, wherein the instructions, when executed by the at least one processing device, cause the at least one processing device to: by the output layer of the neural network: modify the output of the hidden layer with a fourth learned matrix; and modify the topic feature vector with a fifth learned matrix, wherein the first learned matrix, the second learned matrix, the third learned matrix, the fourth learned matrix, and the fifth learned matrix are separate matrices. ### Claim 11 (depends on 9) The system of claim 9, wherein the instructions, when executed by the at least one processing device, cause the at least one processing device to: by the hidden layer of the neural network: perform a first multiplication operation of the word input vector by the first learned matrix to generate a first multiplication output; perform a second multiplication operation of the topic feature vector by the second learned matrix to generate a second multiplication output; perform a third multiplication operation of the time-delayed hidden-state vector by the third learned matrix to generate a third multiplication output; and sum the first multiplication output, the second multiplication output, and the third multiplication output to generate the output of the hidden layer. ### Claim 12 (depends on 8) The system of claim 8, wherein the instructions, when executed by the at least one processing device, cause the at least one processing device to: generate the topic feature vector using a Latent Dirichlet Allocation (LDA) technique; and as subsequent words from the input sequence are processed using the neural network, incrementally generate next topic feature vectors based at least on previous feature vectors. ### Claim 13 (depends on 12) The system of claim 12, wherein the instructions, when executed by the at least one processing device, cause the at least one processing device to: apply a decay factor to previous topic feature vectors for previous words that have already been processed by the neural network. ### Claim 14 (depends on 8) The system of claim 8, wherein the input sequence of words is part of an input document. ### Claim 15 At least one computer readable storage medium storing instructions which, when executed by at least one processing device, cause the at least one processing device to perform acts comprising: receiving a word input vector at an input layer of a neural network, the word input vector representing an individual word from an input sequence of words; receiving a topic feature vector at the input layer of the neural network, the topic feature vector being separate from the word input vector and representing topics expressed in the input sequence of words; using the neural network to generate an output vector at an output layer of the neural network based at least on the word input vector and the topic feature vector, wherein using the neural network includes, by a hidden layer of the neural network: modifying the word input vector using a first learned matrix; and modifying the topic feature vector using a second learned matrix that is separate from the first learned matrix, wherein the output vector represents a word probability given the word input vector and the topic feature vector; and performing a natural language processing operation based at least on the word probability represented by the output vector. ### Claim 16 (depends on 15) The at least one computer readable storage medium of claim 15, wherein using the neural network includes: by the hidden layer of the neural network, modifying a time-delayed hidden-state vector with a third learned matrix, wherein the time-delayed hidden-state vector represents an output of the hidden layer in a prior time instance, wherein the word input vector, the topic feature vector, and the time-delayed hidden-state vector are separate vectors, and wherein the first learned matrix, the second learned matrix, and the third learned matrix are separate matrices. ### Claim 17 (depends on 16) The at least one computer readable storage medium of claim 16, wherein using the neural network includes, by the output layer of the neural network: modifying the output of the hidden layer with a fourth learned matrix; and modifying the topic feature vector with a fifth learned matrix, wherein the first learned matrix, the second learned matrix, the third learned matrix, the fourth learned matrix, and the fifth learned matrix are separate matrices. ### Claim 18 (depends on 16) The at least one computer readable storage medium of claim 16 wherein using the neural network includes, by the hidden layer: performing a first multiplication operation of the word input vector by the first learned matrix to generate a first multiplication output; performing a second multiplication operation of the topic feature vector by the second learned matrix to generate a second multiplication output; performing a third multiplication operation of the time-delayed hidden-state vector by the third learned matrix to generate a third multiplication output; and summing the first multiplication output, the second multiplication output, and the third multiplication output to generate the output of the hidden layer. ### Claim 19 (depends on 15) The at least one computer readable storage medium of claim 15, the acts further comprising: generating the topic feature vector using a Latent Dirichlet Allocation (LDA) technique; and as subsequent words from the input sequence are processed using the neural network, incrementally generating next topic feature vectors based at least on previous feature vectors. ### Claim 20 (depends on 19) The at least one computer readable storage medium of claim 19, wherein said incrementally generating comprises applying a decay factor to previous topic feature vectors for previous words that have already been processed by the neural network.