Human-in-loop robot training and testing system with generative artificial intelligence (ai)

A system for training and/or testing a robot control system, the system comprising:

• a computation device configured to train and/or test the robot control system;
• a data collection device configured to collect feedback data related to one or more human-operated robot tasks that are to be performed by a human data collector;
• one or more sensing devices located in a location of the human data collector;
• and
• a mixed reality device that is worn by the human data collector, the system configured to perform the following:
• provide at the computation device a library of prompt templates that define one or more steps of one or more robot control tasks;
• provide a prompt to one or more generative AI models, the one or more generative AI models generating high-level instructions, the high-level instructions configured to trigger, when executed, a robot to perform the one or more robot control tasks comprising said one or more steps defined by the library of prompt templates;
• convert the high-level instructions into human-operated robot tasks;
• provide the human-operated robot tasks to the mixed reality device worn by the human data collector, the mixed reality device rendering the human-operated robot tasks in a manner that shows the human data collector how to perform the human-operated robot task;
• receive feedback data in response to the human data collector attempting to perform the human-operated robot tasks;
• and
• update the robot control system using the feedback data.