US 2022/0012424 A1Published application

Word and image relationships in combined vector space

AssigneeInternational Business Machines Corporation

InventorsAnkur Tagra, Sudheesh S. Kairali, Vijay Kalangumvathakkal

Life of this patent

Estimating the term…

Loading patent…

Abstract

Embodiments relate to a system, program product, and method for leveraging cognitive systems to facilitate the management of word and image relationships in a combined vector space. More specifically, the system, computer program product, and method disclosed herein facilitate establishing one or more continuous semantic relationships with word vectors, sentence vectors, and image vectors within an image-word combined vector space with N-dimensional coordinates. In general, the word vectors, sentence vectors, and image vectors are generated within their respective domains that are resident within the image-word combined vector space. Such domains of word vectors, sentence vectors, and image vectors include inherent semantic relationships between respective members of each domain that facilitate establishing continuous, formal, virtual relationships between respective word vectors, image vectors, and sentence vectors within the image-word combined vector space.

Claims20

Dependent claims

A computer system comprising:

a server comprising at least one processing device and at least one memory device operably coupled to the at least one processing device;
and
a knowledge base in operable communication with the server, the server configured to:
transmit a plurality of image vectors to a vector space resident within the knowledge base;
transmit a plurality of caption sentence vectors to the vector space, wherein a respective portion of the plurality of caption sentence vectors have one or more semantic relationships with a respective portion of the plurality of image vectors, thereby defining one or more combined image-word semantic relationships;
transmit a plurality of textual content word vectors to the vector space, wherein a respective portion of the plurality of word vectors have one or more semantic relationships with the respective portion of the plurality of image vectors and the respective portion of caption sentence vectors, thereby further defining the one or more combined image-word semantic relationships;
transmit a plurality of textual content sentence vectors to the vector space, wherein a respective portion of the plurality of textual content sentence vectors have one or more semantic relationships with the respective portion of the plurality of image vectors, the respective portion of caption sentence vectors, and the respective portion of the word vectors, thereby further defining the one or more combined image-word semantic relationships;
and
establish one or more continuous semantic relationships through the one or more combined image-word semantic relationships.

6 more elements

A computer program product, comprising:

one or more computer readable storage media;
and
program instructions collectively stored on the one or more computer storage media, the program instructions comprising:
program instructions to transmit a plurality of image vectors to a vector space;
program instructions to transmit a plurality of caption sentence vectors to the vector space, wherein a respective portion of the plurality of caption sentence vectors have one or more semantic relationships with a respective portion of the plurality of image vectors, thereby defining one or more combined image-word semantic relationships;
program instructions to transmit a plurality of textual content word vectors to the vector space, wherein a respective portion of the plurality of word vectors have one or more semantic relationships with the respective portion of the plurality of image vectors and the respective portion of caption sentence vectors, thereby further defining the one or more combined image-word semantic relationships;
program instructions to transmit a plurality of textual content sentence vectors to the vector space, wherein a respective portion of the plurality of textual content sentence vectors have one or more semantic relationships with the respective portion of the plurality of image vectors, the respective portion of caption sentence vectors, and the respective portion of the word vectors, thereby further defining the one or more combined image-word semantic relationships;
and
program instructions to establish one or more continuous semantic relationships through the one or more combined image-word semantic relationships.

6 more elements

A computer-implemented method comprising:

transmitting a plurality of image vectors to a vector space;
transmitting a plurality of caption sentence vectors to the vector space, wherein a respective portion of the plurality of caption sentence vectors have one or more semantic relationships with a respective portion of the plurality of image vectors, thereby defining one or more combined image-word semantic relationships;
transmitting a plurality of textual content word vectors to the vector space, wherein a respective portion of the plurality of word vectors have one or more semantic relationships with the respective portion of the plurality of image vectors and the respective portion of caption sentence vectors, thereby further defining the one or more combined image-word semantic relationships;
transmitting a plurality of textual content sentence vectors to the vector space, wherein a respective portion of the plurality of textual content sentence vectors have one or more semantic relationships with the respective portion of the plurality of image vectors, the respective portion of caption sentence vectors, and the respective portion of the word vectors, thereby further defining the one or more combined image-word semantic relationships;
and
establishing one or more continuous semantic relationships through the one or more combined image-word semantic relationships.

3 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/923,123Filed 8 Jul 2020
Published: 13 Jan 2022
Term ends: Not estimatedThe source records don’t support an estimate.

Classification16 codes

G06F 40/284
Recognition of textual entities: lexical analysis, e.g. tokenisation or collocates
G06F 40/30
Handling natural language data: semantic analysis
G06N 3/0442
Recurrent networks, e.g. Hopfield networks: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
G06N 3/0455
Combinations of networks: auto-encoder networks; Encoder-decoder networks

Family

US 11,501,071 B2Related patent
US 2022/0012424 A1This patent

From the US filing history. Foreign family members aren’t listed.

Timeline

Citations

Open elsewhere

Building an agent or pulling many patents? Use the API or MCP server instead of scraping pages. Get an API key