Method of bidirectional image-text retrieval based on multi-view joint embedding space

#The present invention relates to the field of computer vision, and specifically, to a method of bidirectional image-text retrieval based on a multi-view joint embedding space, which realizes a bi-directional image-text retrieval task by learning to obtain a multi-view joint embedding space. In this method, the data is observed by utilizing different views, to obtain the semantic association relationship of the data under different granularities, and merged using a multi-view merging and sorting method, to obtain more accurate semantic associations and a more accurate retrieval result.

#With the constant development of research in the field of computer vision, a large number of image-text related tasks, such as ImageCaption, DenseCaption, and VQA (Visual Question Answering), have emerged. These tasks all require that the computer fully understands the semantic information of images and texts, and can translate the semantic information of one kind of module data to another module through learning. Therefore, the core problem of this kind of tasks is how to make up the gap between the two different modules at the semantic level, and find the way to communicate the data with two kinds of modules. The data of different modules exist in the heterogeneous space, so it is impossible to measure the semantic similarity directly by calculating the distance between the data. To solve this problem, the current mainstream method is to transform the two kinds of data to obtain isomorphic features, thereby embedding the data into a common space, so that they can be directly compared. Researchers have done a lot of work on which features are used to represent the data and which methods are used to embed the data in a common space.

#Considering that some features derived from deep learning have achieved good results in many tasks in the field of computer vision, a large number of researchers use these features to represent images or text and transform them into isomorphic forms, so that the data can be mapped to a common space to complete the image-text retrieval task. However, these learned features only provide information at the global level of data and lack the description of local information. Therefore, by using only these features to represent data, it is impossible to mine the association between more fine-grained data, such as the association relationship between the region of the image and the phrase in the text. In another method, the image and text are divided into small parts, and mapped into a common space, so as to capture local semantic association information in fine-grained data.

#FIG. 1 is a flow chart of bidirectional image-text retrieval method based on a multi-view joint embedding space in the present invention.

#FIG. 2 is a schematic diagram of learning process in a multi-view joint embedding space in an embodiment of the present invention.

#Where VGG is a 19-layer VGG model, extracting the CNN features of the frame, HGLMM is the Fisher Vector feature of sentence extracted by the Hybrid Gaussian-Laplacian Mixture Model, RCNN is the RCNN feature of the region extracted by using Faster RCNN model, and Parser is a dependency triplet of the extracted phrases by Standford CoreNLP parser. The neural network consists of two fully connected layers, and the two neural networks in each view form a dual-branch neural network.

#The present invention will become apparent from the following detailed description of embodiments and from the accompanying drawings, but not limited to the scope of the present invention in any way.

#The present invention discloses a bidirectional image-text retrieval method based on a multi-view joint embedding space, comprising performing retrieval with reference to a semantic association relationship at a global level and a local level; obtaining the semantic association relationship at the global level and the local level in a frame-sentence view and a region-phrase view, respectively, embedding the image and the sentence in the subspace of the global level, and then obtaining from the semantic association information of frame-sentence view; extracting each region of the image and each phrase of sentence in the region-phrase view and embedding in the subspace of the local level to obtain the semantic association information of the region and the phrase; processing data by a dual-branch neural network in the two views to obtain an isomorphic feature, embedding the same in a common space, and using a constraint condition to reserve an original semantic relationship of the data during training; and merging two semantic association relationships by means of a multi-view merging and sorting method to obtain an more accurate semantic similarity between data.

#We use the data set D={D1, D2, . . . , D|D|} to describe the image-text retrieval problem. In this data set, each document includes a image and a related piece of text, e.g., Di=(Ii, Ti). Each text consists of several sentences, each of which independently describes the matching image. In the frame-sentence view, we use fi to represent an image of the training image Ii, and {si1, si2, . . . , sik} to represent the set of sentences in Ti (k is the number of sentences in text Ti). In the region-phrase view, we use rim to represent the mth region extracted from the frame fi, and pin to represent the nth phrase extracted from the sentence in text Ti. Next we will describe in detail the constraints used in the two views, and the merging and sorting method of the final multi-view merge module.