Conditional data sourcing and curation for generating datasets in deep learning model training

One or more processors coupled to a memory, the one or more processors comprising processing circuitry to perform operations comprising:

• retrieving, by a plurality of processing nodes, a plurality of content items from one or more data sources, wherein each processing node of the plurality of processing nodes retrieves a different subset of the plurality of content items from the one or more data sources;
• applying a first set of filters to metadata associated with the plurality of content items to generate a plurality of filtered content items, wherein the first set of filters comprises a set of conditions associated with the metadata;
• generating similarity scores respectively between content embeddings of content items in the plurality of filtered content items and label embeddings of respective text descriptions for the plurality of filtered content items;
• generating, based on the generated similarity scores and a second set of filters implementing at least a threshold similarity score, a plurality of mappings between the plurality of filtered content items and a plurality of text descriptions for the plurality of filtered content items;
• storing, based on the plurality of mappings, the plurality of filtered content items in respective association with the plurality of text descriptions, as a curated dataset in one or more data stores;
• and
• causing one or more parameters of a machine learning model to be updated using the curated dataset.

A system comprising:

• a data center comprising a plurality of processing nodes, each processing node of the plurality of processing nodes being implemented with one or more processors coupled to a memory to perform operations comprising:
• retrieving, by the plurality of processing nodes, a plurality of content items from one or more data sources, wherein each processing node of the plurality of processing nodes retrieves a different subset of the plurality of content items from the one or more data sources;
• applying a first set of filters to metadata associated with the plurality of content items to generate a plurality of filtered content items, wherein the first set of filters comprises a set of conditions associated with the metadata;
• generating similarity scores respectively between content embeddings of content items in the plurality of filtered content items and label embeddings of respective text descriptions for the plurality of filtered content items;
• generating, based on the generated similarity scores and a second set of filters implementing at least a threshold similarity score, a plurality of mappings between the plurality of filtered content items and a plurality of text descriptions for the plurality of filtered content items;
• storing, based on the plurality of mappings, the plurality of filtered content items in respective association with the plurality of text descriptions, as a curated dataset in one or more data stores;
• and
• causing one or more parameters of a machine learning model to be updated using the curated dataset.