Back to results
US 12,597,236 B2Granted
Fine-tuning joint text-image encoders using reprogramming
AssigneeInternational Business Machines Corporation
InventorsAndrew Geng, Pin-Yu Chen
Life of this patent
Estimating the term…
Abstract
Techniques to fine-tune a joint text-image encoder via model reprogramming. The joint text-image encoder includes an image encoder and a text encoder, which are trained. An image and a caption describing the image are received. A reprogrammed image is generated based on the received image and using a first function. A reprogrammed caption is generated based on the received caption and using a second function. The image encoder and the text encoder are further trained using the reprogrammed image and the reprogrammed caption. One or more parameters for each of the first and second functions are backpropagated to produce, via transfer learning, the fine-tuned joint text-image encoder.
Claims20
A computer-implemented method comprising:
- receiving a joint text-image encoder, which comprises an image encoder and a text encoder, wherein the image encoder and the text encoder have been trained;
- receiving an image and a caption describing the image;
- generating a reprogrammed image based on the received image and via a first trainable image reprogramming function of a plurality of reprogramming functions, wherein the reprogrammed image is applied to the image encoder for extracting reprogrammed image features;
- generating a reprogrammed caption based on the received caption and via a second trainable text reprogramming function of the plurality of reprogramming functions, wherein the reprogrammed caption is applied to the text encoder for extracting reprogrammed text features;
- responsive to generating the reprogrammed image and caption, further training each of the image encoder and the text encoder by one or more computer processors using a respective one of the reprogrammed image and the reprogrammed caption;
- and
- backpropagating a respective one or more parameters for each of the first trainable image reprogramming function and the second trainable text reprogramming function to produce, via transfer learning, a fine-tuned joint text-image encoder based on the received joint text-image encoder, second trainable image reprogramming function” should read “wherein each of the respective one or more parameters comprises a learnable parameter that enables updating the respective first trainable image reprogramming function and second trainable text reprogramming function.
4 more elements
A computer program product comprising:
- a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation comprising:
- receiving a joint text-image encoder, which comprises an image encoder and a text encoder, wherein the image encoder and the text encoder have been trained;
- receiving an image and a caption describing the image;
- generating a reprogrammed image based on the received image and via a first trainable image reprogramming function of a plurality of reprogramming functions, wherein the reprogrammed image is applied to the image encoder for extracting reprogrammed image features;
- generating a reprogrammed caption based on the received caption and via a second trainable text reprogramming function of the plurality of reprogramming functions, wherein the reprogrammed caption is applied to the text encoder for extracting reprogrammed text features;
- responsive to generating the reprogrammed image and caption, further training each of the image encoder and the text encoder using a respective one of the reprogrammed image and the reprogrammed caption;
- and
- backpropagating a respective one or more parameters for each of the first trainable image reprogramming function and the second trainable text reprogramming function to produce, via transfer learning, a fine-tuned joint text-image encoder based on the received joint text-image encoder, second trainable image reprogramming function” should read “wherein each of the respective one or more parameters comprises a learnable parameter that enables updating the respective first trainable image reprogramming function and second trainable text reprogramming function.
5 more elements
A system comprising:
- one or more computer processors;
- and
- a memory containing a program executable by the one or more computer processors to perform an operation comprising:
- receiving a joint text-image encoder, which comprises an image encoder and a text encoder, wherein the image encoder and the text encoder have been trained;
- receiving an image and a caption describing the image;
- generating a reprogrammed image based on the received image and via a first trainable image reprogramming function of a plurality of reprogramming functions, wherein the reprogrammed image is applied to the image encoder for extracting reprogrammed image features;
- generating a reprogrammed caption based on the received caption and via a second trainable text reprogramming function of the plurality of reprogramming functions, wherein the reprogrammed caption is applied to the text encoder for extracting reprogrammed text features;
- responsive to generating the reprogrammed image and caption, further training each of the image encoder and the text encoder using a respective one of the reprogrammed image and the reprogrammed caption;
- and
- backpropagating a respective one or more parameters for each of the first trainable image reprogramming function and the second trainable text reprogramming function to produce, via transfer learning, a fine-tuned joint text-image encoder based on the received joint text-image encoder, second trainable image reprogramming function” should read “wherein each of the respective one or more parameters comprises a learnable parameter that enables updating the respective first trainable image reprogramming function and second trainable text reprogramming function.
7 more elements
Dependencies are inferred from the claim text.