System and method for vision-based control of a reconfigurable soft robotic gripper

A processor implemented method comprising:

• receiving, by one or more hardware processors, a data pertaining to an object grasping environment, wherein the data comprises a current sequence of images pertaining to an object to be grasped, a plurality of sensor data associated with the plurality of flex sensors mounted in a plurality of fingers of a reconfigurable robotic gripper;
• computing, by the one or more hardware processors, a position the object to be grasped based on the current sequence of images pertaining to the object to be grasped;
• simultaneously computing, by the one or more hardware processors, a current magnitude of bending and a current configuration associated with each of the plurality of fingers based on the plurality of sensor data;
• iteratively computing, by the one or more hardware processors, an optimal grasp control signal based on the position of the object to be grasped, the current configuration and the current magnitude of bending associated with the each of the at least three fingers using a trained reward based Reinforcement Learning (RL) model;
• and
• grasping, by the reconfigurable robotic gripper, the object to be grasped based on the optimal grasp control signal, wherein gripper action is performed by actuating a motor associated to each of the plurality of fingers based on the optimal grasp control signal.

One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:

• receiving a data pertaining to an object grasping environment, wherein the data comprises a current sequence of images pertaining to an object to be grasped, a plurality of sensor data associated with the plurality of flex sensors mounted in a plurality of fingers of a reconfigurable robotic gripper;
• computing a position the object to be grasped based on the current sequence of images pertaining to the object to be grasped;
• simultaneously computing a current magnitude of bending and a current configuration associated with each of the plurality of fingers based on the plurality of sensor data;
• iteratively computing an optimal grasp control signal based on the position of the object to be grasped, the current configuration and the current magnitude of bending associated with the each of the at least three fingers using a trained reward based Reinforcement Learning (RL) model;
• and
• grasping the object to be grasped based on the optimal grasp control signal, wherein gripper action is performed by actuating a motor associated to each of the plurality of fingers based on the optimal grasp control signal.