Systems and methods for deep learning model based product matching using multi modal data

A system for generating a list of matching products, wherein each matching product matches a reference product, comprising:

• one or more processors;
• and
• one or more non-transitory, computer-readable storage media for storing program code accessible by the one or more processors, the program code when executed by the one or more processors causes the one or more processors to:
• receive a user query for the reference product;
• determine reference attribute data for the reference product, wherein the reference attribute data is multimodal having at least a first data mode and a second data mode different from the first data mode, wherein each data mode is a type of characterization of the reference product, and wherein each data mode has a modality selected from the group consisting of a text data class, categorical data, a pre-compared engineered feature, audio data, image data, and video data;
• determine a first list of candidate products, based on at least one reference product match signature generated from the reference attribute data by a deep learning product signature model;
• generate a second list of candidate products from the first list of candidate products, wherein at least one given candidate product in the first list of candidate products has multimodal attribute data with at least the first data mode and the second data mode, and wherein for the given candidate product,
• select a deep learning multimodal matching model from a plurality of candidate multimodal matching models, based on a reference category associated with the reference product and a first attribute data item for the given candidate product, wherein each candidate multimodal matching model has been previously trained on a category-specific training dataset, wherein the selected deep learning multimodal matching model comprises at least a first data analyzer for processing data having the first data mode and a second data analyzer for processing data having the second data mode wherein the first data analyzer comprises a first deep learning neural network (DLNN), and wherein the second data analyzer comprises a second DLNN,
• match the given candidate product to the reference product using the selected deep learning multimodal matching model to generate a match score, wherein the match score is computed based on the reference attribute data for the reference product and attribute data for the given candidate product,
• determine whether the match score meets a given threshold, and
• in response to determining that the match score meets the given threshold, add the given candidate product to the second list of candidate products;
• and
• generate the list of matching products by filtering the second list of candidate products to remove outlier candidate products.

A non-transitory, computer-readable storage medium for generating a list of matching products, wherein each matching product matches a reference product, the non-transitory, computer-readable storage medium storing program code accessible by one or more processors, the program code when executed by the one or more processors causes the one or more processors to:

• receive a user query for the reference product;
• determine reference attribute data for the reference product, wherein the reference attribute data is multimodal having at least a first data mode and a second data mode different from the first data mode, wherein each data mode is a type of characterization of the reference product, and wherein each data mode has a modality selected from the group consisting of a text data class, categorical data, a pre-compared engineered feature, audio data, image data, and video data;
• determine a first list of candidate products, based on at least one reference product match signature generated from the reference attribute data by a deep learning product signature model;
• generate a second list of candidate products from the first list of candidate products, wherein at least one given candidate product in the first list of candidate products has multimodal attribute data with at least the first data mode and the second data mode, and wherein for the given candidate product,
• select a deep learning multimodal matching model from a plurality of candidate multimodal matching models, based on a reference category associated with the reference product and a first attribute data item for the given candidate product, wherein each candidate multimodal matching model has been previously trained on a category-specific training dataset, wherein the selected deep learning multimodal matching model comprises at least a first data analyzer for processing data having the first data mode and a second data analyzer for processing data having the second data mode, wherein the first data analyzer comprises a first deep learning neural network (DLNN), and wherein the second data analyzer comprises a second DLNN,
• match the given candidate product to the reference product using the selected deep learning multimodal matching model to generate a match score, wherein the match score is computed based on the reference attribute data for the reference product and attribute data for the given candidate product,
• determine whether the match score meets a given threshold, and
• in response to determining that the match score meets the given threshold, add the given candidate product to the second list of candidate products;
• and
• generate the list of matching products by filtering the second list of candidate products to remove outlier candidate products.