Connecting vision and language using fourier transform
AssigneeSap SE
InventorsStefan Lionar, Tassilo Klein, Moin Nabi
Life of this patent
Estimating the term…
Abstract
A method for text-image integration is provided. The method may include receiving a question related to pairable data comprising text data and image data. Embeddings are generated from the text tokens and image encodings. Embeddings are generated from the text tokens and image encodings. The embeddings include text embeddings and image embeddings. A spectral conversion of the text embeddings and the image embeddings is performed to generate spectral data. The spectral data is processed to extract text-image features. The text-image features are processed to generate inferred answers to the question.
Claims20
A computer-implemented method comprising:
- receiving, by one or more processors, pairable data comprising text data and image data;
- generating, by the one or more processors, text tokens and image encodings from the pairable data;
- generating, by the one or more processors, embeddings from the text tokens and image encodings, the embeddings comprising text embeddings and image embeddings;
- performing, by the one or more processors, a spectral conversion of the text embeddings and image embeddings to generate spectral data;
- processing, by the one or more processors, the spectral data to extract text-image features;
- and
- processing, by the one or more processors, the text-image features to generate inferred answers.
A non-transitory computer-readable storage medium coupled to one or more processors and having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
- receiving pairable data comprising text data and image data;
- generating text tokens and image encodings from the pairable data;
- generating embeddings from the text tokens and image encodings, the embeddings comprising text embeddings and image embeddings;
- performing a spectral conversion of the text embeddings and image embeddings to generate spectral data;
- processing the spectral data to extract text-image features;
- and
- processing the text-image features to generate inferred answers.
A system comprising:
- a computing device;
- and
- a computer-readable storage device coupled to the computing device and having instructions stored thereon which, when executed by the computing device, cause the computing device to perform operations comprising:
- receiving pairable data comprising text data and image data;
- generating text tokens and image encodings from the pairable data;
- generating embeddings from the text tokens and image encodings, the embeddings comprising text embeddings and image embeddings;
- performing a spectral conversion of the text embeddings and image embeddings to generate spectral data;
- processing the spectral data to extract text-image features;
- and
- processing the text-image features to generate inferred answers.
Dependencies are inferred from the claim text.