Identifying items in images using embeddings generated from the images and ranking candidates using a language model

A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to perform steps comprising:

• receiving an image, the image including an unknown item;
• extracting descriptive information about the unknown item from the image based on content of the image;
• segmenting the descriptive information into a bin corresponding to the unknown item by applying a generative artificial intelligence (AI) model to the descriptive information, the bin including one or more attributes of the unknown item the generative AI model identified from the descriptive information;
• generating a target embedding for the unknown item based on the one or more attributes of the unknown item included in the bin;
• selecting, based on the target embedding, a set of candidate embeddings from a database of known embeddings, each candidate embedding corresponding to a known item;
• retrieving, from an item database, a set of attributes for each candidate item;
• generating a ranking of the candidate items by applying the generative AI model to a prompt, the prompt including the attributes of each candidate item, the attributes of the unknown item, and an instruction to rank the candidate items based on matching the attributes of each of the candidate items to the attributes of the unknown item;
• and
• presenting at least a subset of the candidate items based on the ranking the presenting causing the subset of the candidate items to be displayed in a graphical user interface.

A system comprising:

• a processor;
• and
• a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by the processor, cause the processor to perform steps comprising:
• receiving an image, the image including an unknown item;
• extracting descriptive information about the unknown item from the image based on content of the image;
• segmenting the descriptive information into a bin corresponding to the unknown item by applying a generative artificial intelligence (AI) model to the descriptive information, the bin including one or more attributes of the unknown item the generative AI model identified from the descriptive information;
• generating a target embedding for the unknown item based on the one or more attributes of the unknown item included in the bin;
• selecting, based on the target embedding, a set of candidate embeddings from a database of known embeddings, each candidate embedding corresponding to a known item;
• retrieving, from an item database, a set of attributes for each candidate item;
• generating a ranking of the candidate items by applying the generative AI model to a prompt, the prompt including the attributes of each candidate item, the attributes of the unknown item, and an instruction to rank the candidate items based on matching the attributes of each of the candidate items to the attributes of the unknown item;
• and
• presenting at least a subset of the candidate items based on the ranking the presenting causing the subset of the candidate items to be displayed in a graphical user interface.