# US 10,769,374 B1 · Answer selection method for question answering system and the system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,769,374 B1
- Document ID: US10769374B1
- Status: Granted
- Kind: B1
- Application number: 16/831,552
- Filed: 2020-03-26
- Granted: 2020-09-08
- Earliest claimed: 2019-04-24
- Assignees: Honghui Chen
- Applicants: Honghui Chen
- Inventors: Honghui Chen; Taihua Shao; Fei Cai; Danyang Jiang; Junxian Liu; Aimin Luo; Tao Chen; Zhen Shu
- Web page: https://patentgrep.com/patent/US10769374B1

## CPC

- G06F 40/253: Grammatical analysis; Style critique
- G06F 40/30: Semantic analysis
- G06F 17/147: {Discrete orthonormal transforms, e.g. discrete cosine transform, discrete sine transform, and variations therefrom, e.g. modified discrete cosine transform, integer transforms approximating the discrete cosine transform (G06F17/145 takes precedence)}
- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F 40/35: Discourse or dialogue representation
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06N 5/04: Inference or reasoning models
- G06N 3/048: Activation functions
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Disclosed are an answer selection method for a question answering system. With the method and system, a sentence distributed representation of a question and an answer is generated by adopting a hierarchical length-adaptive neural network structure with a purpose of extracting high-quality sentence features by using different neural feature extractors according to a length of an input sentence. First, one word distributed representation is generated for each word in the input sentence by connecting one fixed word embedding and one finely-adjusted word embedding. Next, a BiLSTM-based feature extractor is used for a short sentence and a Transformer-based feature extractor is used for a long sentence. Finally, a sentence vector representation for measuring a correlation between a question and a candidate answer is generated by using an attention pooling layer considering an interaction between a question sentence and an answer sentence.

## Claims

### Claim 1

An answer selection method for a question answering system, comprising the following blocks:
at block A, receiving a question sentence and an answer sentence input by a user, and obtaining a first word vector of each word in each sentence by using a word embedding model obtained through pre-training to calculate a combined word vector in combination with a second word vector obtained by performing fine adjustment for the word embedding model in a network training process;
at block B, obtaining a final vector of each word in the sentence by selecting a valid feature from the connected word vectors;
at block C, when a sentence length is greater or smaller than a preset sentence length threshold, processing the sentence by adopting a different feature extractor according to the preset sentence length threshold to generate a fitted sentence matrix representation containing contextual information;
at block D, obtaining, by an attention pooling layer, an attention similarity of the question sentence and the answer sentence according to the sentence matrix containing contextual information obtained at block C, and performing a length-adaptive distributed representation for the question sentence and the answer sentence by extracting a maximum feature at each dimension of row and column of a similarity matrix respectively to generate a distributed vector representation for the question sentence and the answer sentence respectively; and
at block E, obtaining a correlation of each answer sentence relative to the question sentence by measuring a cosine similarity between the distributed vector representations of the question sentence and the answer sentence, and sorting the answer sentences according to the correlation.

### Claim 2 (depends on 1)

The answer selection method according to claim 1, wherein block A comprises the following blocks:
at block A1, defining the input question sentence and the input answer sentence both as a sentence s with a length being L, obtaining a word embedding model of the t-th word Wt in the sentence s through pre-training in a corpus, and searching for a first word vector rWtfr corresponding to the word therefrom; and
at block A2, performing fine adjustment for parameters of the pre-trained word embedding model in the network training process to adapt the model to a current data set, obtaining the updated word embedding model, and obtaining a second word vector rWtft corresponding to the word therefrom to calculate a combined word vector as rWtC =rWtfr∥rWtft.

### Claim 3 (depends on 2)

The answer selection method according to claim 2, wherein block B comprises the following blocks:
at block B1, deploying a hidden layer, and obtaining a final vector representation of each word in the sentence by selecting a valid feature from the connected word vectors based on
rWt=tanh (Wh·rWtC+bh),
wherein h refers to a word representation with a memory in a particular direction, WhϵD×D and bhϵD×1 both refer to hidden layer parameters, refers to a set of real numbers, D×D refers to a real number matrix with dimensions being D×D, D×1 refers to a real number matrix with dimensions being D×1, and D refers to a dimension of rWt, and
at block B2, obtaining a sentence distributed matrix representation containing no contextual information based on Rs=(rW1rW2 . . . rWL) by performing blocks A1, A2 and B1 for all words in all sentences.

### Claim 4 (depends on 1)

The answer selection method according to claim 1, wherein at block C, two feature extractors, i.e., a BiLSTM-based feature extractor and a Transformer-based feature extractor, are deployed for each input sentence, and a fitted sentence distributed matrix representation is generated by performing addition for the sentence distributed matrixes generated with two feature extractors.

### Claim 5 (depends on 4)

The answer selection method according to claim 4, wherein at block C, each feature extractor is defined to have one binary activation parameter, and the binary activation parameters are set as follows:
{
flag
lstm
=
1
⁢
⁢
and
⁢
⁢
flag
tsfm
=
0
,
(
L
≤
L
threshold
)
flag
lstm
=
0
⁢
⁢
and
⁢
⁢
flag
tsfm
=
1
,
(
L
>
L
threshold
)
wherein flaglstm refers to the binary activation parameter of the BiLSTM-based feature extractor,flagtsfm refers to the binary activation parameter of the Transformer-based feature extractor, and Lthreshold refers to a sentence length threshold, and
the inputs of different sentences based on the feature extractors are as follows:
Rslstm=Rs·flaglstm
Rstsfm=Rs·flagtsfm
wherein, Rslstm refers to an input of the BiLSTM-based feature extractor, and Rstsfm refers to an input of the Transformer-based feature extractor.

### Claim 6 (depends on 5)

The answer selection method according to claim 5, wherein at block C, when the sentence length L is smaller than the threshold Lthreshold, the method further comprises:
a specific operation formula for the representation input of the t-th word in the sentence being as follows:
{
f
t
=
sigmoid
⁡
(
W
f
·
r
w
t
lstm
+
U
f
·
h
t
-
1
+
b
f
)
i
t
=
sigmoid
⁡
(
W
i
·
r
w
t
lstm
+
U
i
·
h
t
-
1
+
b
i
)
=
tanh
⁡
(
W
c
·
r
w
t
lstm
+
U
c
·
h
t
-
1
+
b
c
)
C
t
=
f
t
·
C
t
-
1
+
i
t
·
o
t
=
sigmoid
⁡
(
W
o
·
r
w
t
lstm
+
U
o
·
h
t
-
1
+
b
o
)
h
t
=
o
t
·
tanh
⁡
(
C
t
)
r
w
t
short
=
h
t
→


⁢
h
t
←
Wherein rWtlstm refers to a representation input of the t-th word, i, o and f refer to an input gate, an output gate and a forget gate in the BiLSTM respectively, {tilde over (C)} and C refer to a retained total memory and a memory retained for the current word respectively, h refers to a word representation with a memory in a particular direction where its dimension is H, {right arrow over (h)} and refer to forward and backward word representations with memories respectively, WϵH×D, UϵH×H and bϵH×1 are all network parameters, Wf, Uf and bf are network parameters of the forget gate, Wi, Ui and bi are network parameters of the input gate,Wo, Uoand bo are network parameters of the output gate, WC, UC and bC are network parameters of the retained total memory, and sigmoid and tanh are both common activation functions in a neural network; and
obtaining a final word representation rWtshort each word in a short sentence by splicing the word representations in two directions, wherein the BiLSTM-based vector representations constitute a sentence matrix representation Rsshort containing contextual information as follows:
Rsshort=(rW1shortrW2short . . . rWLshort).

### Claim 7 (depends on 5)

The answer selection method according to claim 5, wherein at block C, when the sentence length L is greater than the threshold Lthreshold, feature extraction is performed for the sentence based on the following calculation formula:
{
f
⁡
(
R
s
tsfm
)
a
i
=
O
a
i
T
·
tanh
⁡
(
W
a
i
·
R
s
tsfm
+
U
a
i
·
R
s
tsfm
)
R
s
a
i
=
R
s
tsfm
·
soft
⁢
⁢
max
⁡
(
f
⁡
(
R
s
tsfm
)
a
i
T
d
R
s
tsfm
)
R
s
long
=
R
s
a
1
⁢

R
s
a
2

⁢
⁢
…
⁢

⁢
R
s
a
n
=
(
r
w
1
long
⁢
r
w
2
long
⁢
⁢
…
⁢
⁢
r
w
L
long
)
wherein f(Rstsfm)ai refers to self-attention similarity of the i-th self-attention mechanism in a multi-head self-attention mechanism of a Transformer neural network; Rsai refers to a sentence matrix representation effected by the i-th self-attention, O, W and U are self-attention parameters;dRstsfm refers to a dimension of each word vector in the input sentence matrix representation, √{square root over (dRstsfm)} aims to compress a softmax function, and Rslong refers to a Transformer-based sentence matrix representation containing contextual information.

### Claim 8 (depends on 1)

The answer selection method according to claim 1, wherein at block D, a specific operation formula used by the attention pooling layer for the question sentence and the answer sentence is as follows:
{
G
=
R
Q
T
·
U
·
R
A
v
q
=
R
Q
·
soft
⁢
⁢
max
⁡
(
Column
⁢
⁢
Max
⁡
(
G
)
)
v
a
=
R
A
·
soft
⁢
⁢
max
⁡
(
Row
⁢
⁢
Max
⁡
(
G
)
)
wherein G refers to an attention similarity of a question and an answer, RQ refers to a sentence matrix representation containing contextual information for the question sentence,RQT refers to a transposition of RQ, RA refers to a sentence matrix representation containing contextual information for the answer sentence, and U is an attention parameter;ColumnMax(*) and RowMax(*)are functions of taking maximum values of each column and each row of the matrix respectively.

### Claim 9 (depends on 1)

The answer selection method according to claim 1, wherein at block E, the neural network is trained by minimizing a sorting loss of the following candidate answers in such a way that the neural network reaches an optimal training stage:
loss=max {0, m−cos(vq,va+)+cos(vq,va−)}
wherein m is a preset interval, vq refers to a distributed vector representation of the question sentence, va refers to a distributed vector representation of the answer sentence, a+ refers to a positive answer in a training instance, and a− refers to a negative answer in the training instance.

### Claim 10

An answer selection system for a question answering system, comprising a client and a server, wherein the client is connected with the server through a wireless network, wherein the server comprises the following modules:
in module A, receiving a question sentence and an answer sentence input by a user, and obtaining a first word vector of each word in each sentence by using a word embedding model obtained through pre-training to calculate a combined word vector in combination with a second word vector obtained by performing fine adjustment for the word embedding model in a network training process;
in module B, obtaining a final vector of each word in the sentence by selecting a valid feature from the connected word vectors;
in module C, when a sentence length is greater or smaller than a preset sentence length threshold, processing the sentence by adopting a different feature extractor according to the preset sentence length threshold to generate a fitted sentence matrix representation containing contextual information;
in module D, obtaining, by an attention pooling layer, an attention similarity of the question sentence and the answer sentence according to the sentence matrix containing contextual information obtained at block C, and performing a length-adaptive distributed representation for the question sentence and the answer sentence by extracting a maximum feature at each dimension of row and column of a similarity matrix respectively to generate a distributed vector representation for the question sentence and the answer sentence respectively; and
in module E, obtaining a correlation of each answer sentence relative to the question sentence by measuring a cosine similarity between the distributed vector representations of the question sentence and the answer sentence, and sorting the answer sentences according to the correlation,
wherein the module A comprises the following units:
in unit A1, defining the input question sentence and the input answer sentence both as a sentence s with a length being L, obtaining a word embedding model of the t-th word Wt in the sentence s through pre-training in a corpus, and searching for a first word vector rWtfr corresponding to the word therefrom; and
in unit A2, performing fine adjustment for parameters of the pre-trained word embedding model in the network training process to adapt the model to a current data set, obtaining the updated word embedding model, and obtaining a second word vector r Wtft corresponding to the word therefrom to calculate a combined word vector as rWtC=rWtfr ∥rWtft,
wherein the module B comprises the following units:
in unit B1, deploying a hidden layer and obtaining a final vector representation of each word in the sentence by selecting a valid feature from the connected word vectors based on rWt=tanh (Wh·rWtC+bh),
wherein h refers to a word representation with a memory in a particular direction, WhϵD×D and bhϵD×1 both refer to hidden layer parameters, refers to a set of real numbers, D×D refers to a real number matrix with dimensions being D×D, D×1 refers to a real number matrix with dimensions being D×1, and D refers to a dimension of rWt,and
in unit B2, obtaining a sentence distributed matrix representation containing no contextual information based on Rs=(rW1rW2 . . . rWL) by performing blocks A1, A2 and B1 for all words in all sentences.