Prompt tuning for zero-shot compositional learning in machine learning systems

An apparatus comprising:

• at least one processing device configured to:
• obtain an image, a set of attribute labels, and a set of object labels;
• and
• perform prompt tuning of a pre-trained vision-language model comprising a first textual encoder, a second textual encoder, and a vision encoder, the pre-trained vision-language model trained during the prompt tuning to select one of the attribute labels and one of the object labels that match content contained in the image;
• wherein, to perform the prompt tuning, the at least one processing device is configured, for each of multiple attribute label-object label pairs, to:
• generate object textual features associated with the object label of the attribute label-object label pair using the first textual encoder;
• generate attribute textual features associated with the attribute label of the attribute label-object label pair using the second textual encoder;
• and
• generate image features associated with the image using the vision encoder;
• wherein the at least one processing device is configured to combine intermediate outputs from initial layers of the first textual encoder, the second textual encoder, and the vision encoder to generate layer-specific learnable prompt tokens that are appended to inputs of specified layers in the first textual encoder, the second textual encoder, and the vision encoder during the prompt tuning.

A method comprising:

• obtaining an image of an object;
• obtaining a set of attribute labels and a set of object labels;
• and
• selecting one of the attribute labels and one of the object labels associated with the object using a vision-language model comprising a first textual encoder, a second textual encoder, and a vision encoder;
• wherein selecting the one of the attribute labels and the one of the object labels associated with the object comprises:
• generating object textual features associated with each of the object labels using the first textual encoder;
• generating attribute textual features associated with each of the attribute labels using the second textual encoder;
• and
• generating image features associated with the image using the vision encoder;
• wherein one or more layers in the first textual encoder, one or more layers in the second textual encoder, and one or more layers in the vision encoder are each associated with a layer-specific multi-modal shared prompt that is concatenated to an input for the layer.