Scalable pipeline for machine learning-based base-variant grouping

A method being implemented via execution of computing instructions configured to run on one or more processors and stored at one or more non-transitory computer-readable media, the method comprising:

• generating text embeddings of the candidate items using a first pre-trained Siamese neural network and a first contrastive loss function;
• and
• converting the text embeddings to first N-dimensional feature embeddings, wherein the first N-dimensional feature embeddings comprise encoded vector representations used to compute similarity between the candidate items for variant grouping;
• generating an adjacency list for candidate items using a distance threshold with a maximum allowable neighbor distance equal to a Siamese model cut-off, wherein the distance threshold is determined using a union logic for pairs of the candidate items;
• loading data for the candidate items in the adjacency list;
• generating graphs of the candidate items in the adjacency list, wherein nodes of the graphs represent the candidate items and wherein edges of the graphs represent predicted variant neighbor links between pairs of the candidate items;
• determining, using breakdown logic, first graphs of the graphs that exceed a predetermined size;
• building hierarchy dendrograms of nested subclusters of the first graphs;
• determining cut-off values based on p-th percentiles of density for the first graphs;
• identifying recommended variant groups of the candidate item in the nested subclusters of the hierarchy dendrograms below the cut-off values;
• and
• presenting the recommended variant groups on electronic media used for online ordering.

A non-transitory computer-readable medium storing instructions, wherein the instructions, upon execution by a processor, cause the processor to perform operations comprising:

• generating text embeddings of the candidate items using a first pre-trained Siamese neural network and a first contrastive loss function;
• and
• converting the text embeddings to first N-dimensional feature embeddings, wherein the first N-dimensional feature embeddings comprise encoded vector representations used to compute similarity between the candidate items for variant grouping;
• generating an adjacency list for candidate items using a distance threshold with a maximum allowable neighbor distance equal to a Siamese model cut-off, wherein the distance threshold is determined using a union logic for pairs of the candidate items;
• loading data for the candidate items in the adjacency list;
• generating graphs of the candidate items in the adjacency list;
• determining, using breakdown logic, first graphs of the graphs that exceed a predetermined size;
• building hierarchy dendrograms of nested subclusters of the first graphs using respective linkage distances between item embeddings of the candidate items of pairs of the candidate items of the first graphs;
• determining cut-off values based on p-th percentiles of density for the first graphs;
• identifying recommended variant groups of the candidate item in the nested subclusters of the hierarchy dendrograms below the cut-off values;
• and
• presenting the recommended variant groups on electronic media used for online ordering.