Text-based image retrieval
AssigneeAdobe Inc.
InventorsHyunjae Kim, Seunghyun Yoon, Trung Huu Bui, Handong Zhao, Quan Tran, Franck Dernoncourt
Life of this patent
Estimating the term…
Abstract
A method, apparatus, non-transitory computer readable medium, and system for media processing include obtaining a text prompt describing content, generating, using a multi-modal encoder, a text embedding based on the text prompt, and obtaining an image depicting the content based on the text embedding. The multi-modal encoder is trained to encode image descriptions based on a similarity between a caption of a training image and a paraphrase of the caption.
Claims20
A method for media processing, comprising:
- obtaining a text prompt describing content;
- generating, using a text encoder of a multi-modal encoder, a text embedding in a multi-modal embedding space based on the text prompt, wherein the text encoder is trained to encode image descriptions in the multi-modal embedding space based on a similarity between a caption of a training image and a paraphrase of the caption and wherein the caption is different from the paraphrase;
- and
- obtaining an image depicting the content based on the text embedding.
A method for training a machine learning model, comprising:
- obtaining a training set comprising a training image, a caption of the training image, and a paraphrase of the caption;
- encoding, using an image encoder of a multi-modal encoder, the training image to obtain an image embedding in a multi-modal embedding space;
- encoding, using a text encoder of the multi-modal encoder, the caption and the paraphrase to obtain a caption embedding and a paraphrase embedding, respectively, in the multi-modal embedding space;
- and
- training the text encoder of the multi-modal encoder by updating parameters of the text encoder based on a first similarity between the image embedding and the caption embedding and a second similarity between the caption embedding and the paraphrase embedding.
A system for media processing, comprising:
- at least one processor;
- at least one memory storing instructions executable by the at least one processor;
- and
- a multi-modal encoder comprising a text encoder comprising encoding parameters stored in the at least one memory, the text encoder configured to generate a text embedding in a multi-modal embedding space based on a text prompt, wherein the text encoder is trained to encode image descriptions in the multi-modal embedding space based on a similarity between a caption of a training image and a paraphrase of the caption and wherein the caption is different from the paraphrase.
Dependencies are inferred from the claim text.