# US 12,118,301 B2 · Method and apparatus for generating hidden state in recurrent neural network for language processing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,301 B2 - Document ID: US12118301B2 - Status: Granted - Kind: B2 - Application number: 17/332,318 - Filed: 2021-05-27 - Published: 2021-09-16 - Granted: 2024-10-15 - Earliest claimed: 2019-04-17 - Assignees: Tencent Technology (Shenzhen) Company Limited - Applicants: Tencent Technology (Shenzhen) Company Limited - Inventors: Fandong Meng; Jinchao Zhang; Jie Zhou - Web page: https://patentgrep.com/patent/US12118301B2 ## CPC - G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30) - G06F 18/10: Pre-processing; Data cleansing - G06F 18/25: Fusion techniques - G06F 40/237: Lexical tools - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G06F 40/30: Semantic analysis - G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/047: Probabilistic or stochastic networks - G06N 3/048: Activation functions - G06N 3/08: Learning methods - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A method and apparatus for generating a hidden state in a recurrent neural network for language processing are provided. The method includes: generating regional word vectors of at least two dimensions of a target word vector inputted at a first moment; combining the regional word vectors to obtain combined regional word vectors of at least two dimensions; performing aggregation transformation processing on the combined regional word vectors based on a feedforward neural network, to obtain an aggregated word vector corresponding to the target word vector; and generating a target hidden state corresponding to the target word vector based on the aggregated word vector. ## Claims ### Claim 1 A method for generating a hidden state in a recurrent neural network for sequentially inputted language processing, applied to a computer device, the method comprising: generating regional word vectors of at least two dimensions of a target word vector inputted at a first moment based on at least two first weight matrices and a second hidden state corresponding to a second word vector inputted at a second moment being earlier than the first moment in time, each of the first weight matrices being used for generating a corresponding regional word vector; combining the regional word vectors in a preset regional vector combination manner to obtain combined regional word vectors of the at least two dimensions, wherein the preset regional vector combination manner includes a graph convolution network combination manner for processing the regional word vectors as vector nodes in a graph convolution network or a capsule network combination manner for processing the regional word vectors as capsules in a capsule network; performing aggregation transformation processing on the combined regional word vectors based on a feedforward neural network, to obtain an aggregated word vector corresponding to the target word vector; and generating a target hidden state corresponding to the target word vector based on the aggregated word vector. ### Claim 2 (depends on 1) The method according to claim 1, wherein the generating the regional word vectors of the at least two dimensions of the target word vector based on the at least two first weight matrices and the second hidden state corresponding to the second word vector inputted at the second moment being earlier than the first moment in time comprises: concatenating the target word vector and the second hidden state to obtain a concatenated word vector; and generating a regional word vector matrix according to the concatenated word vector and the at least two first weight matrices, the regional word vector matrix comprising the regional word vectors of the at least two dimensions. ### Claim 3 (depends on 1) The method according to claim 1, wherein the combining the regional word vectors in the graph convolution network combination manner to obtain combined regional word vectors of the at least two dimensions comprises: determining edge weights between the regional word vectors, each edge weight corresponding to two of the regional word vectors; generating an adjacency matrix corresponding to the regional word vectors according to the determined edge weights; respectively adding up the edge weights of each dimension in the adjacency matrix to obtain a degree matrix; and generating the combined regional word vectors of the at least two dimensions based on the adjacency matrix and the degree matrix. ### Claim 4 (depends on 3) The method according to claim 3, wherein the generating the combined regional word vectors of at least two dimensions based on the adjacency matrix and the degree matrix comprises: determining a regional word vector matrix corresponding to the regional word vectors; obtaining a second weight matrix used for generating a combined regional word vector matrix; and generating the combined regional word vector matrix according to the adjacency matrix, the degree matrix, the regional word vector matrix, and the second weight matrix, the combined regional word vector matrix comprising the regional word vectors of the at least two dimensions. ### Claim 5 (depends on 1) The method according to claim 1, wherein the combining the regional word vectors in the capsule network combination manner to obtain combined regional word vectors of the at least two dimensions comprises: performing operations of: determining at least two prediction vectors corresponding to the regional word vectors according to a third weight matrix used for generating the combined regional word vectors; determining at least two prior probability logarithms corresponding to the regional word vectors; determining coupling coefficients corresponding to the regional word vectors according to the at least two prior probability logarithms; and generating candidate combined regional word vectors of the at least two dimensions based on the coupling coefficients and the at least two prediction vectors; and performing an iterative calculation on the candidate combined regional word vectors by performing again the operations starting from determining the at least two prior probability logarithms corresponding to the regional word vectors, stopping the iterative calculation when a preset iteration condition is met, and determining the candidate combined regional word vectors of the at least two dimensions when the iterative calculation is stopped as the combined regional word vectors of the at least two dimensions. ### Claim 6 (depends on 5) The method according to claim 5, wherein the determining at least two prior probability logarithms corresponding to the regional word vectors further comprises: determining scalar products of the candidate combined regional word vectors and the corresponding prediction vectors; and adding up the scalar products and the corresponding prior probability logarithms, to obtain updated prior probability logarithms corresponding to the regional word vectors. ### Claim 7 (depends on 1) The method according to claim 1, wherein the performing aggregation transformation processing on the combined regional word vectors based on a feedforward neural network, to obtain an aggregated word vector corresponding to the target word vector comprises: transforming the combined regional word vectors based on the feedforward neural network, to obtain transformed combined regional word vectors; concatenating the transformed combined regional word vectors, to obtain a concatenated word vector; and performing linear transformation on the concatenated word vector, to obtain the aggregated word vector corresponding to the target word vector. ### Claim 8 (depends on 7) The method according to claim 7, wherein the transforming the combined regional word vectors based on the feedforward neural network, to obtain transformed combined regional word vectors comprises: performing linear transformation on the combined regional word vectors according to a fourth weight matrix and a first bias vector, to obtain temporary word vectors corresponding to the combined regional word vectors; respectively selecting maximum vector values in the temporary word vectors and a vector threshold; and respectively performing linear transformation on the maximum vector values according to a fifth weight matrix and a second bias vector, to obtain the transformed combined regional word vectors. ### Claim 9 (depends on 1) The method according to claim 1, wherein the aggregated word vector comprises a first aggregated word vector and a second aggregated word vector, the first aggregated word vector being different from the second aggregated word vector; and the generating a target hidden state corresponding to the target word vector based on the aggregated word vector comprises: determining a candidate hidden state corresponding to the target word vector based on the first aggregated word vector and a corresponding first activation function; determining a gating parameter corresponding to the target word vector based on the second aggregated word vector and a corresponding second activation function; and generating the target hidden state corresponding to the target word vector according to the candidate hidden state, the gating parameter, and the historical hidden state of the historical word vector at the previous moment. ### Claim 10 An apparatus for generating a hidden state in a recurrent neural network for language processing, the apparatus comprising a memory and a processor, the memory storing a computer program, the computer program, when executed by the processor, causing the processor to: generate regional word vectors of at least two dimensions of a target word vector inputted at a first moment based on at least two first weight matrices and a second hidden state corresponding to a second word vector inputted at a second moment being earlier than the first moment in time, each of the first weight matrices being used for generating a corresponding regional word vector; combine the regional word vectors in a preset regional vector combination manner to obtain combined regional word vectors of the at least two dimensions, wherein the preset regional vector combination manner includes a graph convolution network combination manner for processing the regional word vectors as vector nodes in a graph convolution network or a capsule network combination manner for processing the regional word vectors as capsules in a capsule network; perform aggregation transformation processing on the combined regional word vectors based on a feedforward neural network, to obtain an aggregated word vector corresponding to the target word vector; and generate a target hidden state corresponding to the target word vector based on the aggregated word vector. ### Claim 11 (depends on 10) The apparatus according to claim 10, wherein the processor is further configured to: concatenate the target word vector and the second hidden state to obtain a concatenated word vector; and generate a regional word vector matrix according to the concatenated word vector and the at least two first weight matrices, the regional word vector matrix comprising the regional word vectors of the at least two dimensions. ### Claim 12 (depends on 10) The apparatus according to claim 10, wherein the processor is further configured to: determine edge weights between the regional word vectors, each edge weight corresponding to two of the regional word vectors; generate an adjacency matrix corresponding to the regional word vectors according to the determined edge weights; respectively add up the edge weights of each dimension in the adjacency matrix to obtain a degree matrix; and generate the combined regional word vectors of the at least two dimensions based on the adjacency matrix and the degree matrix. ### Claim 13 (depends on 12) The apparatus according to claim 12, wherein the processor is further configured to: determine a regional word vector matrix corresponding to the regional word vectors; obtain a second weight matrix used for generating a combined regional word vector matrix; and generate the combined regional word vector matrix according to the adjacency matrix, the degree matrix, the regional word vector matrix, and the second weight matrix, the combined regional word vector matrix comprising the regional word vectors of the at least two dimensions. ### Claim 14 (depends on 10) The apparatus according to claim 10, wherein the processor is further configured to: perform operations of: determining at least two prediction vectors corresponding to the regional word vectors according to a third weight matrix used for generating the combined regional word vectors; determining at least two prior probability logarithms corresponding to the regional word vectors; determining coupling coefficients corresponding to the regional word vectors according to the at least two prior probability logarithms; and generating candidate combined regional word vectors of the at least two dimensions based on the coupling coefficients and the at least two prediction vectors; and perform an iterative calculation on the candidate combined regional word vectors by performing again the operations starting from determining the at least two prior probability logarithms corresponding to the regional word vectors, stop the iterative calculation when a preset iteration condition is met, and determine candidate combined regional word vectors of the at least two dimensions when the iterative calculation is stopped as the combined regional word vectors of the at least two dimensions. ### Claim 15 (depends on 14) The apparatus according to claim 14, wherein the processor is further configured to: determine scalar products of the candidate combined regional word vectors and the corresponding prediction vectors; and add up the scalar products and the corresponding prior probability logarithms, to obtain updated prior probability logarithms corresponding to the regional word vectors. ### Claim 16 (depends on 10) The apparatus according to claim 10, wherein the processor is further configured to: transform the combined regional word vectors based on the feedforward neural network, to obtain transformed combined regional word vectors; concatenate the transformed combined regional word vectors, to obtain a concatenated word vector; and perform linear transformation on the concatenated word vector, to obtain the aggregated word vector corresponding to the target word vector. ### Claim 17 (depends on 16) The apparatus according to claim 16, wherein the processor is further configured to: perform linear transformation on the combined regional word vectors according to a fourth weight matrix and a first bias vector, to obtain temporary word vectors corresponding to the combined regional word vectors; respectively select maximum vector values in the temporary word vectors and a vector threshold; and respectively perform linear transformation on the maximum vector values according to a fifth weight matrix and a second bias vector, to obtain the transformed combined regional word vectors. ### Claim 18 A non-transitory computer-readable storage medium, storing a computer program, the computer program, when executed by a processor, causing the processor to perform: generating regional word vectors of at least two dimensions of a target word vector inputted at a first moment based on at least two first weight matrices and a second hidden state corresponding to a second word vector inputted at a second moment being earlier than the first moment in time, each of the first weight matrices being used for generating a corresponding regional word vector; combining the regional word vectors in a preset regional vector combination manner to obtain combined regional word vectors of the at least two dimensions, wherein the preset regional vector combination manner includes a graph convolution network combination manner for processing the regional word vectors as vector nodes in a graph convolution network or a capsule network combination manner for processing the regional word vectors as capsules in a capsule network; performing aggregation transformation processing on the combined regional word vectors based on a feedforward neural network, to obtain an aggregated word vector corresponding to the target word vector, and generating a target hidden state corresponding to the target word vector based on the aggregated word vector.