Connecting vision and language using fourier transform
AssigneeSap SE
InventorsStefan Lionar, Tassilo Klein, Moin Nabi
Life of this patent
Estimating the term…
Abstract
A method for text-image integration is provided. The method may include receiving a question related to pairable data comprising text data and image data. Embeddings are generated from the text tokens and image encodings. Embeddings are generated from the text tokens and image encodings. The embeddings include text embeddings and image embeddings. A spectral conversion of the text embeddings and the image embeddings is performed to generate spectral data. The spectral data is processed to extract text-image features. The text-image features are processed to generate inferred answers to the question.
Claims14
A computer-implemented method comprising:
- receiving, by one or more processors, pairable data comprising text data and image data;
- generating, by the one or more processors, text tokens and image encodings from the pairable data;
- generating, by the one or more processors, embeddings from the text tokens and image encodings, the embeddings comprising text embeddings and image embeddings;
- performing, by the one or more processors, a spectral conversion of the text embeddings, the image embeddings, and a sentinel token that captures a pooled representation of the text embeddings and the image embeddings to generate spectral data, wherein the spectral conversion comprises applying a Fourier transform to convert the text embeddings, the image embeddings, and the sentinel token from a time domain to a spectral domain;
- processing, by the one or more processors, the spectral data to extract text-image features, wherein the text-image features comprise text information and image information that are blended together;
- and
- processing, by the one or more processors, the text-image features in part by predicting correlation values for each of the text-image features to generate inferred answers answering text questions to given reference images, wherein the inferred answers are generated based on particular text-image features having a correlation value exceeding a set threshold.
A non-transitory computer-readable storage medium coupled to one or more processors and having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
- receiving pairable data comprising text data and image data;
- generating text tokens and image encodings from the pairable data;
- generating embeddings from the text tokens and image encodings, the embeddings comprising text embeddings and image embeddings;
- performing a spectral conversion of the text embeddings, the image embeddings, and a sentinel token that captures a pooled representation of the text embeddings and the image embeddings to generate spectral data, wherein the spectral conversion comprises applying a Fourier transform to convert the text embeddings, the image embeddings, and the sentinel token from a time domain to a spectral domain;
- processing the spectral data to extract text-image features, wherein the text-image features comprise text information and image information that are blended together;
- and
- processing the text-image features in part by predicting correlation values for each of the text-image features to generate inferred answers answering text questions to given reference images, wherein the inferred answers are generated based on particular text-image features having a correlation value exceeding a set threshold.
A system comprising:
- a computing device;
- and
- a computer-readable storage device coupled to the computing device and having instructions stored thereon which, when executed by the computing device, cause the computing device to perform operations comprising:
- receiving pairable data comprising text data and image data;
- generating text tokens and image encodings from the pairable data;
- generating embeddings from the text tokens and image encodings, the embeddings comprising text embeddings and image embeddings;
- performing a spectral conversion of the text embeddings, the image embeddings, and a sentinel token that captures a pooled representation of the text embeddings and the image embeddings to generate spectral data, wherein the spectral conversion comprises applying a Fourier transform to convert the text embeddings, the image embeddings, and the sentinel token from a time domain to a spectral domain;
- processing the spectral data to extract text-image features, wherein the text-image features comprise text information and image information that are blended together;
- and
- processing the text-image features in part by predicting correlation values for each of the text-image features to generate inferred answers answering text questions to given reference images, wherein the inferred answers are generated based on particular text-image features having a correlation value exceeding a set threshold.
Dependencies are inferred from the claim text.