On-robot data collection

A system comprising:

• a computing device comprising one or more processors, the computing device configured to perform a method comprising:
• operating a robot to generate sensor data about an environment, to navigate within the environment, and to perform tasks in the environment;
• determining, by a controller of the robot, that a first set of sensor data about the environment requires manual annotation, wherein determining that the first set of sensor data requires manual annotation comprises at least one of i) applying the first set of sensor data to a first model to generate (1) an output representation of the first set of sensor data and (2) an estimate of a confidence of the output and determining that the estimate of the confidence of the output is less than a threshold confidence, ii) determining that the robot has failed to perform a first task, wherein the first set of sensor data represents the environment during the performance of the first task or immediately after failure to perform the first task, or iii) determining that a first mode of sensor data within the first set of sensor data does not match a second mode of sensor data within the first set of sensor data;
• and responsive to determining that the first set of sensor data requires manual annotation, transmitting, from a transmitter of the robot to a remote server, an indication of the first set of sensor data and an indication that the first set of sensor data requires manual annotation.

A non-transitory computer readable medium comprising program instructions executable by at least one processor to cause the at least one processor to perform a method comprising:

• operating a robot to generate sensor data about an environment, to navigate within the environment, and to perform tasks in the environment;
• determining, by a controller of the robot, that a first set of sensor data about the environment requires manual annotation, wherein determining that the first set of sensor data requires manual annotation comprises at least one of i) applying the first set of sensor data to a first model to generate (1) an output representation of the first set of sensor data and (2) an estimate of a confidence of the output and determining that the estimate of the confidence of the output is less than a threshold confidence, ii) determining that the robot has failed to perform a first task, wherein the first set of sensor data represents the environment during the performance of the first task or immediately after failure to perform the first task, or iii) determining that a first mode of sensor data within the first set of sensor data does not match a second mode of sensor data within the first set of sensor data;
• and
• responsive to determining that the first set of sensor data requires manual annotation, transmitting, from a transmitter of the robot to a remote server, an indication of the first set of sensor data and an indication that the first set of sensor data requires manual annotation.