Hybrid machine learning-based systems and methods for training an object picking robot with real and simulated performance data

A system for training an object picking robot with real and simulated grasp performance data, comprising:

• one or more memory devices;
• and one or more processors in communication with the one or more memory devices and the object picking robot, wherein the one or more processors are programmed to:
• (a) assign a plurality of grasp locations on an object based on known or estimated physical properties of the object;
• (b) perform a first simulation experiment for the robot grasping the object using a first set of the plurality of assigned grasp locations;
• (c) evaluate, based on a first set of simulation data from the first simulation experiment, a simulated object grasp quality of the robot grasping for each of the first set of assigned grasp locations;
• (d) determine a first set of candidate grasp locations on the object based on data representative of a simulated grasp quality obtained in (c);
• (e) evaluate, based on a first set of sensor data from a first actual experiment for the robot grasping the object using each of the first set of candidate grasp locations, an actual object grasp quality of the robot grasping for the each of the first set of candidate grasp locations;
• and
• (f) for the each of the first set of candidate grasp locations, determine a convergence of the actual object grasp quality and the simulated object grasp quality based on:
• the first set of grasp quality sensor data from the first actual experiment, and the data representative of the simulated grasp quality.

A non-transitory computer-readable storage medium storing processor-executable instructions for training an object picking robot with real and simulated grasp performance data, wherein, when executed by one or more processors, the processor-executable instructions cause the one or more processors to:

• (a) assign a plurality of grasp locations on an object based on known or estimated physical properties of the object;
• (b) perform a first simulation experiment for the robot grasping the object using a first set of the plurality of assigned grasp locations;
• (c) evaluate, based on a first set of simulation data from the first simulation experiment, a simulated object grasp quality of the robot grasping for each of the first set of assigned grasp locations;
• (d) determine a first set of candidate grasp locations on the object based on data representative of a simulated grasp quality obtained in (c);
• and
• (e) evaluate, based on a first set of grasp quality sensor data from a first actual experiment for the robot grasping the object using each of the first set of candidate grasp locations, an actual object grasp quality of the robot grasping for the each of the first set of candidate grasp locations, and
• (f) for the each of the first set of candidate grasp locations, determine a convergence of the actual object grasp quality and the simulated object grasp quality based on:
• the first set of grasp quality sensor data from the first actual experiment, and the data representative of the simulated grasp quality.