Visual dialogue method and system

A visual dialogue method, comprising:

• obtaining original input data, wherein the original input data comprises current image data and a new question, and the new question is related to the current image data;
• preprocessing text data and image data in the original input data to obtain a text feature sequence and a visual feature sequence, respectively;
• using a Visual Dialog (VisDial) dataset to construct a text corpus;
• obtaining text sequence knowledge based on the visual feature sequence and the text corpus;
• constructing a sparse scene graph based on the visual feature sequence;
• performing a data fusion on the text feature sequence, the visual feature sequence, the text sequence knowledge, and the sparse scene graph to obtain a data fusion result;
• and
• obtaining dialogue content of the new question by decoding based on the data fusion result;
• obtaining a text data feature in the text corpus;
• calculating a similarity between the text data feature and the visual feature sequence to obtain the text sequence knowledge;
• wherein the step of performing the data fusion on the text feature sequence, the visual feature sequence, the text sequence knowledge, and the sparse scene graph to obtain the data fusion result comprises:
• obtaining a first attention result based on the text feature sequence and the new question;
• obtaining a second attention result based on the text sequence knowledge;
• cascading the first attention result and the second attention result to obtain a cascading result;
• obtaining a third attention result based on the visual feature sequence and the second attention result;
• performing a graph convolution on the sparse scene graph to obtain a graph convolution result;
• and
• obtaining the data fusion result based on the cascading result, the third attention result, and the graph convolution result;
• wherein the step of obtaining the first attention result based on the text feature sequence and the new question comprises:
• performing a sentence-level attention guidance on the text feature sequence by using the new question to obtain an attention feature;
• filtering the attention feature by using a sigmoid activation function to obtain a sentence-level sequential representation of potential knowledge;
• obtaining a word-level sequential representation of the potential knowledge by calculating a dot-product of the attention feature and the sigmoid activation function based on a word-level question feature of the new question;
• and
• obtaining the first attention result based on the attention feature and the word-level sequential representation of the potential knowledge.