Multimodal machine learning image and text combined search method

A system, comprising:

• a computing device;
• and
• a computer-readable storage device coupled to the computing device and having instructions stored thereon which, when executed by the computing device, cause the computing device to perform operations for automatic index creation for relational database systems, the operations comprising:
• receiving from a repository a plurality of items, wherein each item has an image and a textual description associated therewith;
• generating a first image feature vector, for a first item, by processing a respective image using a first machine learning model;
• generating a first textual feature vector, for the first item, by processing a respective textual description using a second machine learning model;
• combining, for the first item, the first image feature vector for the first item and the first textual feature vector for the first item, to generate a first combined feature vector for the first item;
• generating, for the first item, a first similarity list of similar items, wherein a first similar item is included in the first similarity list based on a similarity between the first image feature vector for the first item and a similar image feature vector for the similar item;
• generating, for the first item, a second similarity list of similar items, wherein a second similar item is included in the second similarity list based on a similarity between the first text feature vector for the first item and a similar text feature vector for the second similar item;
• generating, for the first item, a third similarity list of similar items, wherein a third similar item is included in the third similarity list based on a similarity between the first combined feature vector for the first item and a similar combined feature vector for the third similar item;
• combining the first similarity list for the first item, the second similarity list for the first item, and the third similarity list for the first item to generate a combined similarity list for the first item by:
• identifying a first set of similar items for the first item that are in only one of the first similarity list for the first item, the second similarity list for the first item, or the third similarity list for the first item, and
• including the first set of similar items and respective similarity values for the first set of similar items in the combined similarity list of similar items for the first item;
• receiving a request for information for the first item;
• retrieving the combined similarity list of similar items for the first item;
• and
• providing the combined similarity list of similar items for the first item in response to the request.

A non-transitory computer-readable storage medium coupled to one or more processors and having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations for automatic index creation for relational database systems, the operations comprising:

• receiving from a repository a plurality of items, wherein each item has an image and a textual description associated therewith;
• generating a first image feature vector, for a first item, by processing a respective image using a first machine learning model;
• generating a first textual feature vector, for the first item, by processing a respective textual description using a second machine learning model;
• combining, for the first item, the first image feature vector for the first item and the first textual feature vector for the first item, to generate a first combined feature vector for the first item;
• generating, for the first item, a first similarity list of similar items, wherein a first similar item is included in the first similarity list based on a similarity between the first image feature vector for the first item and a similar image feature vector for the similar item;
• generating, for the first item, a second similarity list of similar items, wherein a second similar item is included in the second similarity list based on a similarity between the first text feature vector for the first item and a similar text feature vector for the second similar item;
• generating, for the first item, a third similarity list of similar items, wherein a third similar item is included in the third similarity list based on a similarity between the first combined feature vector for the first item and a similar combined feature vector for the third similar item;
• combining the first similarity list for the first item, the second similarity list for the first item, and the third similarity list for the first item to generate a combined similarity list for the first item by:
• identifying a first set of similar items for the first item that are in only one of the first similarity list for the first item, the second similarity list for the first item, or the third similarity list for the first item, and
• including the first set of similar items and respective similarity values for the first set of similar items in the combined similarity list of similar items for the first item;
• receiving a request for information for the first item;
• retrieving the combined similarity list of similar items for the first item;
• and
• providing the combined similarity list of similar items for the first item in response to the request.