Evaluating robot learning

A system comprising:

• one or more computers;
• and
• one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
• receiving, by the one or more computers, classification examples from a plurality of remote devices over a communication network, wherein each classification example includes (i) an embedding generated by a remote device running a machine learning model based on sensor data captured by the remote device, wherein the embedding is a vector whose values correspond to features of an object and (ii) a classification corresponding to the embedding;
• assigning, by the one or more computers, quality scores to the classification examples, wherein the quality score assigned to a particular classification example including a particular embedding and a particular classification is based on a level of similarity using a vector distance comparison of the vector of the particular embedding with other vectors of other embeddings sharing the classification;
• determining, by the one or more computers, a training data set, wherein determining the training data set comprises selecting, by the one or more computers, a subset of the classification examples to include in the training data set based on the quality score assigned to each classification example of the classification examples;
• training, by the one or more computers, a version of the machine learning model using the training data set;
• and
• providing the version of the machine learning model to the plurality of remote devices.

One or more non-transitory computer-readable media storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

• receiving, by the one or more computers, classification examples from a plurality of remote devices over a communication network, wherein each classification example includes (i) an embedding generated by a remote device running a machine learning model based on sensor data captured by the remote device, wherein the embedding is a multidimensional vector whose values correspond to features of an object and (ii) a classification corresponding to the embedding;
• assigning, by the one or more computers, quality scores to the classification examples, wherein the quality score assigned to a particular classification example including a particular embedding and a particular classification is based on a level of similarity using a vector distance comparison of the vector of the particular embedding with other vectors of other embeddings sharing the classification;
• determining, by the one or more computers, a training data set, wherein determining the training data set comprises selecting, by the one or more computers, a subset of the classification examples to include in the training data set based on the quality score assigned to each classification example of the classification examples;
• training, by the one or more computers, a version of the machine learning model using the training data set;
• and
• providing the version of the machine learning model to the plurality of remote devices.