Vision-language model with an ensemble of experts

A computer-implemented method, comprising:

• receiving multi-modal labels corresponding to an image, wherein the multi-modal labels are generated by two or more pre-trained domain-specific neural network models;
• mapping the multi-modal labels to a fixed length sequence of multi-modal features;
• and
• processing the image and the fixed length sequence of multi-modal features by a vision encoder and language decoder to predict text corresponding to the image, wherein the fixed length of the sequence of multi-modal features is unchanged in response to removing the multi-modal labels generated by one of the two or more pre-trained domain-specific neural network models or adding additional multi-modal labels generated by an additional pre-trained domain-specific neural network model.

A computer-implemented method, comprising:

• receiving multi-modal labels corresponding to an image, wherein the multi-modal labels are generated by two or more pre-trained domain-specific neural network models;
• mapping the multi-modal labels to a fixed length sequence of multi-modal features;
• and
• processing the image and the fixed length sequence of multi-modal features by a vision encoder and language decoder to predict text corresponding to the image, wherein the vision encoder comprises an adaptor configured to:
• down-project vision features from a first dimension to a second dimension;
• apply a non-linear operation to the down-projected vision features to produce smoothed vision features;
• and
• up-project the smoothed vision features to the first dimension.

A computer-implemented method, comprising

• receiving multi-modal labels corresponding to an image, wherein the multi-modal labels are generated by at least one pre-trained domain-specific neural network model that performs a vision task and a different pre-trained domain-specific neural network model that performs the vision task;
• mapping the multi-modal labels to a fixed length sequence of multi-modal features;
• and
• processing the image and the fixed length sequence of multi-modal features by a vision encoder and language decoder to predict text corresponding to the image.

A system, comprising:

• a memory that stores an image;
• and
• a processor that is connected to the memory, wherein the processor is configured to:
• receive multi-modal labels corresponding to the image, wherein the multi-modal labels are generated by two or more pre-trained domain-specific neural network models;
• map the multi-modal labels to a fixed length sequence of multi-modal features;
• and
• process the image and the fixed length sequence of multi-modal features by a vision encoder and language decoder to predict text corresponding to the image, wherein the fixed length of the sequence of multi-modal features is unchanged in response to removing the multi-modal labels generated by one of the two or more pre-trained domain-specific neural network models or adding additional multi-modal labels generated by an additional pre-trained domain-specific neural network model.

A non-transitory computer-readable media storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:

• receiving multi-modal labels corresponding to an image, wherein the multi-modal labels are generated by two or more pre-trained domain-specific neural network models;
• mapping the multi-modal labels to a fixed length sequence of multi-modal features;
• and
• processing the image and the fixed length sequence of multi-modal features by a vision encoder and language decoder to predict text corresponding to the image, wherein the fixed length of the sequence of multi-modal features is unchanged in response to removing the multi-modal labels generated by one of the two or more pre-trained domain-specific neural network models or adding additional multi-modal labels generated by an additional pre-trained domain-specific neural network model.