Real-time recommendation of data labeling providers

A data processing method comprising:

• receiving, from a user computer, a user dataset to be labeled and labeling instructions;
• receiving and storing in a data schema of a provider metrics database, real-time and historical performance values and metrics for three or more labeling partners in a computer-implemented labeling marketplace, the real-time historical performance values and metrics being updated in the provider metrics database in real time as the three or more labeling partners complete labeling tasks for training data for machine learning models and transmit the real-time historical performance values and metrics from the three or more labeling partners to the system;
• determining, based on the user dataset and the labeling instructions, a type of data of the user dataset and a type of labeling task represented in the labeling instructions;
• identifying, from among the three or more labeling partners, a subset of at least three labeling partners that are able to label the type of data provided by the user computer or to perform the type of labeling task requested by the customer and generating and transmitting, to the user computer, presentation instructions which when rendered using the user computer cause displaying at the user computer a graphical user interface comprising an ordered list of the at least three labeling partners, the at least three labeling partners being ordered in the graphical user interface based on highest accuracy, fastest time, and/or lowest price based on queries to the provider metrics database that yield result sets of values for accuracy, time, and/or price for all the three or more labeling partners in the marketplace;
• querying real-time and historical performance values and metrics stored in the provider metrics database for each labeling partner in the subset of one or more labeling partners to select a selected labeling partner optimal to perform the requested labeling task;
• transmitting the user data and labeling instructions to the selected labeling partner;
• receiving a labeled user dataset from the selected labeling partner after the selected labeling partner has conducted labeling of the user dataset;
• evaluating a quality of the labeled user dataset and updating a record in the provider metrics database associated with the selected labeling partner to specify the quality;
• transmitting the labeled user data to the user computer.