Identification of items in an image and recommendation of similar enterprise products

A method, comprising:

• transforming a first dataset of images by adding item definitions and annotated attributes, wherein the first dataset of images depicts items of a same type or similar type as items included within a retail item catalog;
• creating a first training set comprising the transformed first dataset of images;
• training, at a performance monitoring and model training application, an object detection model using the first training set;
• transforming a second dataset of images by assigning attributes, wherein the second dataset of images depicts retail items within a retail item catalog;
• creating a second training set comprising the transformed second dataset of images;
• training, at the performance monitoring and model training application, an image similarity model using the second training set;
• receiving an image at the object detection system, the image depicting one or more unlabeled retail items;
• identifying, via the object detection model, at least one area of the image, the area being a sub-region of the image and including an item from among the one or more unlabeled retail items;
• generating a bounding box around each identified area;
• when more than one area is identified, determining whether an overlap exists between bounding boxes;
• when an overlap between bounding boxes is not detected, cropping the image to extract each identified retail item;
• when an overlap between bounding boxes is detected, determining a confidence score of a partial object, wherein the confidence score is based at least in part on the number of pixels within the bounding box;
• when the confidence score exceeds a predetermined level, cropping the image to extract at least one partial object of the one or more identified retail items and;
• receiving, at a category and embedding data application, the cropped image segment;
• predicting a category of the item within the area using the image similarity model;
• generating a set of embedding vectors, using numerical representations based on pixel information, associated with the area within the boundary, the embedding vectors being representative of the item;
• receiving, at a visual similarity search application, a plurality of sets of catalog item embedding vectors, wherein each set of catalog item embedding vectors from among the plurality of sets of catalog item embedding vectors is associated with an image of a retail item that is included within the retail item catalog;
• comparing the set of embedding vectors representative of the item to each of the plurality of sets of catalog item embedding vectors to determine a set of images of retail items included within the retail item catalog that are visually similar to the area of the image including the item, the set of images of retail items being associated with a corresponding set of retail items within the retail item catalog;
• based on the comparison, assigning a similarity rank to each retail item of the set of retail items;
• determining a plurality of item attributes associated with the item;
• comparing the retail item attributes associated with each item of the set of retail items to corresponding item attributes from among the plurality of item attributes of the item to obtain a plurality of attribute similarity scores, each of the retail item attributes having assigned to it a predetermined attribute weighting;
• based on (a) the comparison of the retail item attributes to the corresponding item attributes of the item and (b) the predetermined weighted attribute values, assigning, at a weighted average scoring application, one or more attribute scores to each retail item of the set of retail items;
• assigning a final ranking to each retail item of the set of the retail items, based on the similarity rank and the one or more attribute scores assigned to each retail item;
• identifying one or more recommended retail items of the set of retail items based on the final ranking;
• calculating performance metrics relating to the performance of the object detection model and the image similarity model;
• and
• when at least one performance metric of the object detection model or of the image similarity model falls below a predetermined threshold, automatically performing a re-training process on the respective model using updated datasets containing images of new items to the retail item catalog.

An item matching system for a retail website comprising:

• an apparel item matching system implemented on a computing system, the apparel item matching system being configured to:
• transform a first dataset of images by adding item definitions and annotated attributes, wherein the first dataset of images depicts items of a same type or similar type as items included within a retail item catalog;
• create a first training set comprising the transformed first dataset of images;
• train, at a performance monitoring and model training application, an object detection model using the first training set;
• transform a second dataset of images by assigning attributes, wherein the second dataset of images depicts retail items within a retail item catalog;
• create a second training set comprising the transformed second dataset of images;
• train, at the performance monitoring and model training application, an image similarity model using the second training set;
• receive an image at the object detection model, the image depicting one or more unlabeled retail items;
• identify, by the object detection model, at least one area of the image, the area being a sub-region of the image and including an item from among the one or more unlabeled retail items;
• generate a bounding box around each identified area;
• when more than one area is identified, determine whether an overlap exists between bounding boxes;
• when an overlap between bounding boxes is not detected, crop the image to extract each identified retail item;
• when an overlap between bounding boxes is detected, determine a confidence score of a partial object, wherein the confidence score is based at least in part on the number of pixels within the bounding box;
• when the confidence score exceeds a predetermined level, crop the image to extract at least one partial object of the one or more identified retail items;
• receive, at a category and embedding data application, the cropped image segment;
• predict a category of the item within the area using the image similarity model;
• generate a set of embedding vectors, using numerical representations based on pixel information, associated with the area within the boundary, the embedding vectors being representative of the item;
• receive, at a visual similarity search application, a plurality of sets of catalog item embedding vectors, wherein each set of catalog item embedding vectors from among the plurality of sets of catalog item embedding vectors is associated with an image of a retail item that is included within the retail item catalog;
• compare the set of embedding vectors representative of the item to each of the plurality of sets of catalog item embedding vectors to determine a set of images of retail items included within the retail item catalog that are visually similar to the area of the image including the item, the set of images of retail items being associated with a corresponding set of retail items within the retail item catalog;
• based on the comparison, assign a similarity rank to each retail item of the set of retail items;
• determine a plurality of item attributes associated with the item;
• compare the retail item attributes associated with each item of the set of retail items to corresponding item attributes from among the plurality of item attributes of the item to obtain a plurality of attribute similarity scores, each of the retail item attributes having assigned to it a predetermined attribute weighting;
• based on (a) the comparison of the retail item attributes to the corresponding item attributes of the item and (b) the predetermined weighted attribute values, assign, at a weighted average scoring application, one or more attribute scores to each retail item of the set of retail items;
• assign a final ranking to each retail item of the set of the retail items, based on the similarity rank and the one or more attribute scores assigned to each retail item;
• identify one or more recommended retail items of the set of retail items based on the final ranking;
• calculate performance metrics relating to the performance of the object detection model and the image similarity model;
• and
• when at least one performance metric of the object detection model or of the image similarity model falls below a predetermined threshold, automatically perform a re-training process on the respective model using updated datasets containing images of new items to the retail item catalog;
• and
• a retail web server communicatively connected to the computing system on which the apparel item matching system is implemented, the retail web server being configured to generate a user interface including the image, the recommendation, and an option to purchase the recommended retail item.