US 2026/0094424 A1 · SYSTEM AND METHOD FOR ADAPTING VISION-LANGUAGE MODELS WITH HYPERNETWORKS

Abstract

A computer-implemented method and system relate to training a vision language model (VLM), which includes at least an image encoder and a text encoder. The VLM is trained with data pairs, where a data pair includes (i) image data of a digital image and (ii) text data describing that corresponding image data. The text encoder generates text embeddings using the text data. A hypernetwork generates at least a subset of parameters for the image encoder using the text embeddings. The image encoder generates image embeddings using the image data while at least the subset of parameters is applied. A loss is minimized between the image embeddings and the text embeddings. The VLM and the hypernetwork are updated using the loss. The image encoder is relatively small-scale and employable on a resource-constrained device, such as an edge device.

Claims20

Dependent claims

A computer-implemented method for training a machine learning model that includes an image encoder and a text encoder, the computer-implemented method comprising:

receiving data pairs that include image data and text data, each text data describing the corresponding image data of a digital image;
generating, via the text encoder, text embeddings based on the text data;
generating, via a neural network, at least a subset of parameters for the image encoder using the text embeddings;
generating, via the image encoder, image embeddings based on pixels of the image data while the subset of parameters are applied;
minimizing a loss between the image embeddings and the text embeddings;
and
updating the machine learning model and the neural network using the loss.

4 more elements

Dependencies are inferred from the claim text.

System and method for adapting vision-language models with hypernetworks

Life of this patent

System and method for adapting vision-language models with hypernetworks

Life of this patent

Abstract

Claims20