US 2023/0401243 A1 · MULTI-TURN DIALOGUE SYSTEM AND METHOD BASED ON RETRIEVAL

Abstract

The multi-turn dialogue system based on retrieval includes the following modules: a representation module, a matching module, an aggregation module and a prediction module; the multi-turn dialogue method based on retrieval includes the following steps: (1) by a representation module, converting each turn of dialogue into a cascade vector of the dialogue, and converting a candidate answer into a cascade vector of the candidate answer; (2) by a matching module, dynamically absorbing context information based on a global attention mechanism, and calculating a matching vector; (3) by aggregation module, obtaining a short-term dependence information sequence and a long-term dependence information sequence; (4) by a prediction module, calculating the matching score of the context and candidate answer involved in the matching; (5) selecting a candidate answer with the highest matching score as a correct answer.

A multi-turn dialogue system based on retrieval, comprising the following modules:

a representation module, configured to represent each turn of dialogue as a cascade vector Eu of a word level vector and a character level vector, and represent a candidate answer r as a cascade vector Er of a word level vector and a character level vector;
a matching module, formed by stacking L global interaction layers of IOI models, and configured to take the cascade vector of the dialogue and the cascade vector of the candidate answer output by the representation module as an input, dynamically absorb context information based on a global attention mechanism, and recursively calculate a k-th layer self-attention dialogue representation Ūk, a k-th layer self-attention candidate answer representation {circumflex over (R)}k, a k-th layer mutual attention dialogue representation Ūk, a k-th mutual attention candidate answer representation Rk, a k-th layer dialogue synthesis representation Uk, and a k-th layer candidate answer synthesis representation Rk, by the following formulas, to obtain a matching vector (v1, . . . , v1):
Ûk=fcatt(Uk-1,Uk-1,C)
{circumflex over (R)}k=fcatt(Rk-1,Rk-1,C)
Ūk=fcatt(Uk-1,Rk-1,C)
Rk=fcatt(Rk-1,Uk-1,C)
Ũk=[Uk-1,Ûk,Ūk,Uk-1⊙Ūk]
{tilde over (R)}k=[Rk-1,{circumflex over (R)}k,Rk,Rk-1⊙Rk]
Uk=max(0,WhŨk-1+bh)
Rk=max(0,Wh{tilde over (R)}k-1+bh)+Rk-1
in the formulas, Uk-1∈m×d and Rk-1∈n×d represent inputs of a k-th global interaction layer, wherein m and n represent the number of words contained in a current turn of dialogue and the number of words contained in the candidate answer, respectively, and inputs of a first global interaction layer are U0=Eu, R0=Er;
Wh∈4d×d and bh are training parameters;
an operator ⊙ represents a multiplication of elements;
d represents a dimension of a vector;
C∈lc×d represents context obtained by cascading contents of all l turns of dialogues;
all l turns of dialogues contain lc words, and C can be obtained by cascading word level vectors of the lc words;
in the formulas, fcatt( ) represents the global attention mechanism, which is specifically defined as follows:
fcatt(Q,K,C)={tilde over (Q)}+FNN({tilde over (Q)}),
where, FNN({tilde over (Q)})=max(0,{tilde over (Q)}Wf+bf)Wgbg;
W{f,g}∈d×d and b{f,g} are trainable parameters;
Q and {circumflex over (Q)} are mixed using a residual connection to obtain {tilde over (Q)}, wherein {circumflex over (Q)} is calculated according to the following formula:
{circumflex over (Q)}=S(Q,K,C)·K
where, Q∈nq×d represents a query sequence, K∈nk×d represents a key sequence, wherein nq and nk represent the number of words, S(Q,K,C)∈nq×nk represents a similarity of Q and K in the context C;
S(Q, K, C) is calculated according to the following formulas:
S
⁡
(
Q
,
K
,
C
)
i
,
j
=
exp
⁡
(
e
i
,
j
)
∑
k
=
1
n
k
⁢
e
i
,
k
e
i
,
j
=
W
b
⁢
tanh
⁡
(
W
c
⁢
Q
i
+
W
d
⁢
K
j
+
W
e
⁢
C
i
1
+
C
j
k
2
)
where, W{b,c,d,e} are trainable parameters, Ciq represents an i-th row of Cq, and its physical meaning is fusion context information related to an i-th word in the query sequence Q;
Cjk represents a j-th row of Ck, and its physical meaning is fusion context information related to a j-th word of the key sequence K;
Cq∈nq×d and Ck∈nk×d represent context information compression vector fusing the query vector Q and context information compression vector fusing the key vector K, respectively, and are calculated according to the following formulas:
Cq=softmax(QWaCT)·C
Ck=softmax(KWaCT)·C
Wa∈d×d are training parameters;
and
extract a d dimension matching vector vi from a matching image Mi of an i-th turn of dialogue by a convolutional neural network, wherein matching vectors from the first to l-th turn of dialogues are represented by (v1, . . . , vl);
the matching image Mi of the i-th turn of dialogue is calculated according to the following formula:
Mi=Mi,self⊕Mi,interaction⊕Mi,enhanced
where, Mi∈ml×n×3, ⊕ is a cascading operation, mi is the number of words contained in the i-th turn of dialogue ui;
Mi,self, Mi,interaction and Mi,enhanced are calculated according to the following formulas:
M
i
,
self
=
U
^
i
L
·
(
R
^
L
)
⊤
d
M
i
,
interaction
=
U
_
i
L
·
(
R
_
L
)
⊤
d
M
i
,
enhanced
=
U
i
L
·
(
R
L
)
⊤
d
.
an aggregation module composed of one RNN network and one Transformer network, and configured to:
receive the matching vector (v1, . . . , vl) output by the matching module, process the matching vector by the RNN network to obtain a short-term dependence information sequence (h1, . . . , hl), and process the matching vector by the Transformer network to obtain a long-term dependence information sequence (g1, . . . , gl);
wherein a specific calculation process of the short-term dependence information sequence (h1, . . . , hl) is:
obtaining/hidden layer state vectors by processing the matching vector (v1, . . . , vl) through a GRU model, wherein an i-th hidden layer state is:
hj=GRU(vj,hj-1)
where, h0 is initialized randomly;
a specific calculation process of the long-term dependence information sequence (g1, . . . , gl) is:
(g1, . . . ,gl)=MultiHead(Q,K,V)
where, Q=VmWQ, K=VmWK, V=VmWV, wherein WQ, WK and WV are training parameters;
MultiHead( ) represents a multi-head attention function;
Vm=(v1, . . . , vl);
a prediction module, configured to calculate a matching score of the context c and the candidate answer r involved in the matching according to the short-term dependence information sequence (h1, . . . , hl) and long-term dependence information sequence (g1, . . . , gl) output by the aggregation module, which includes:
calculating
g
^
i
=
max
⁡
(
0
,
W
[
(
g
i
-
h
i
)
⊙
(
g
i
-
h
i
)
g
i
⊙
h
i
]
+
b
)
to obtain (ĝ1, . . . , ĝl), wherein ⊕ represents the multiplication of elements;
then inputting (ĝ1, . . . , ĝl) into a GRU model, to obtain:
{tilde over (g)}i=GRU(ĝi,{tilde over (g)}i-1)
wherein {tilde over (g)}0 is initialized randomly;
a final hidden layer state of the GRU model is {tilde over (g)}1;
calculating the matching score of the context c and the candidate answer r involved in matching based on {tilde over (g)}l:
g(c,r)=σ({tilde over (g)}1·wo+bo)
where, σ(·) represents a sigmoid function, wo and bo are training parameters.

206 more elements

Multi-turn dialogue system and method based on retrieval

Life of this patent

Multi-turn dialogue system and method based on retrieval

Life of this patent

Abstract

Claims4