Multimodal data heterogeneous transformer-based asset recognition method, system, and device

A multimodal data heterogeneous Transformer-based asset recognition system, comprising:

• an information collection module, a model building module, a text data encoder, an image data encoder, a text data classifier, an image data classifier, a matching information extraction module, a multimodal space projector, a similarity calculation module, a content generation module, a discriminative fusion module, and an asset class information outputting module, wherein:
• the information collection module collects various-modal information of an asset, comprising text information and image information;
• the model building module builds an A Lite Bidirectional Encoder Representations from Transformers, ALBERT, model, a Vision Transformer, ViT, model, and a Contrastive Language-Image Pre-Training, CLIP, model;
• the text data encoder uses the ALBERT model to extract a text information feature, and uses a multilayer Transformer encoder to learn a context relation in a text sequence, and it comprises:
• preprocessing the text information;
• converting the preprocessed text information into a vector representation;
• adding an identifier to indicate a start or an end;
• performing padding and truncating;
• randomly replacing part of the texts with [MASK] tokens;
• and by a Masked Language Modeling, MLM, model, performing inferential prediction;
• generating a token embedding vector Etoken, a segment embedding vector Eseg, and a position embedding vector Epos;
• and representing a generated embedding by:
• E=Etoken∥Eseg∥Epos
• wherein the ∥ denotes concatenation;
• randomly initializing a token embedding matrix and selecting a corpus for training, wherein the training comprises updating values in the embedding matrix to fit the corpus;
• setting a memorized token embedding vector upon termination of the training to be a final embedding vector;
• learning a paragraph containing a word based on the segment embedding vector;
• and learning a relative position of a word based on the position embedding vector;
• feeding the generated embedding into a multilayer perceptron to obtain a vector Eobj;
• feeding the vector Eobj into the Transformer encoder to generate a presence vector Epresent, denoted as:
• Epresent=Transformerencoder(Eobj)
• wherein the Transformer, denotes the Transformer encoder;
• the text data classifier connects an output of the ALBERT model to a fully connected layer, and outputs final classification information, and it comprises:
• passing the presence vector Epresent through the fully connected layer MLPclass and a softmax function for classification to obtain an recognition type as:
• type=softmax(MLPclass(Epresent));
• the image data encoder uses the ViT model to extract an image information feature:
• divide the image information into tokens;
• use a Transformer encoder to capture and learn content information from the dividing of the image information;
• the image data classifier uses a classification head to map an image feature to class information;
• the matching information extraction module uses the CLIP model to extract image-text matching information feature:
• build an image with matching text description sample pair;
• encode the image information and the text information to obtain an image feature representation vector and a text feature representation vector;
• the multimodal space projector linearly project the image feature representation vector and the text feature representation vector into a multimodal space;
• the similarity calculation module calculates a similarity between two modalities to obtain a matching degree between the image information and the text information;
• the content generation module uses the CLIP model to generate asset void information;
• the discriminative fusion module uses different channels to apply asset type recognition to information in different modalities, output classification information from the different channels, and discriminatively fuse the classification information from the different channels with the matching degree between the image information and the text information obtained by the CLIP model, and it comprises:
• for an asset having both an image and a text, performing a discriminative fusion training, comprising:
• obtaining final feature embedding vectors from the text channel and the image channel, respectively, wherein distances between the feature embedding vectors of the matching text and image as in respective modal spaces are denoted as:
• Diss=(Epresenti−Eimagei)
• Disn=(Epresenti−Eimagej)i≠j
• wherein the Epresenti denotes a text feature embedding vector, the Eimagei denotes an image feature embedding vector, the Diss denotes a distance between the feature embedding vectors of image and text having matching information, and the Disn denotes a distance between the feature embedding vectors of non-matching image and text;
• wherein, in different modalities, distances between embedding vectors representing different information is denoted as:
• Dis(ep1,ep2)=Dis(ei1,ei2)
• Dis(ep1,ep2)=ep1−ep2
• Dis(ei1,ei2)=ei1−ei2
• using the matching degree output from the CLIP model as an accumulation term to build a loss for the discriminative fusion training, denoted as:
• Losscritic=min α(−Σlog σ(Diss−Disn))+β(Dis(eii,eij)+Dis(ep1,ep2))+γSim
• where α, β, γ are automatically learned and generated for different datasets, σ is a sigmoid activation function, and Sim denotes the matching degree;
• after the training, obtaining the discriminatively fused embedding vector representation Efinal, and passing the same through a softmax classifier for classification to obtain the final asset class information, denoted as:
• Classfinal=softmax(critic(Epresent,Eimage,Sim))
• wherein the Epresent denotes the text channel feature embedding vector, the Eimage denotes the image channel feature embedding vector, and the Classfinal denotes the final asset class information;
• and
• the asset class information outputting module outputs the final asset class information.