# US 12,766,706 B2 · System and method for vision-based control of a reconfigurable soft robotic gripper

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,766,706 B2
- Document ID: US12766706B2
- Status: Granted
- Kind: B2
- Application number: 19/205,659
- Filed: 2025-05-12
- Published: 2025-11-13
- Granted: 2026-10-20
- Earliest claimed: 2024-05-13
- Assignees: Tata Consultancy Services Limited
- Applicants: Tata Consultancy Services Limited
- Inventors: Nijil George; Rolif Lima; Vighnesh Vatsal
- Web page: https://patentgrep.com/patent/US12766706B2

## CPC

- B25J 9/1612: {characterised by the hand, wrist, grip control}
- B25J 9/104: {with cables, chains or ribbons}
- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- B25J 9/1697: {Vision controlled systems}
- B25J 13/088: {with position, velocity or acceleration sensors}
- B25J 15/103: {for gripping the object in three contact points}
- B25J 15/12: with flexible finger members
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})

## Abstract

The present disclosure provides a vision based control of a reconfigurable soft robotic gripper. Conventional methods lack an intelligent control method which automatically adapts the configuration of the gripper based on the target object. The present disclosure includes a robotic gripper with a rigid palm and three soft fingers attached to the palm. Each finger is separately actuated using three separate motors. Out of the three fingers, one is fixed, and other two can move relative to the base, the motion of the movable fingers is actuated by motor(s). Each finger has multiple sensors to detect the magnitude of bending. A camera is attached to the base of the gripper, facing the object to be grasped. The camera image and the sensors measurements are given to a control unit and the control unit in turn gives the control signal to the actuator motors to suit the object to be grasped.

## Claims

### Claim 1

A reconfigurable robotic gripper, comprising:
at least three fingers and a rigid palm, wherein the at least three fingers are attached to the rigid palm, wherein each of the at least three fingers is fully three-dimensional (3D) printed using a soft material,
wherein each of the at least three fingers is actuated independently using one or more cables connected to a shaft of a motor associated with each of the at least three fingers,
wherein, one among the at least three fingers is fixed, and the other fingers are movable relative to a base,
wherein motion of the fingers that are movable is actuated by the associated motor,
wherein each finger from among the at least three fingers comprises a plurality of flex sensors to detect a magnitude of bending, and
wherein an image capturing device capable of capturing a sequence of images is attached to the base of the gripper, facing the object to be grasped.

### Claim 2

A processor implemented method comprising:
receiving, by one or more hardware processors, a data pertaining to an object grasping environment, wherein the data comprises a current sequence of images pertaining to an object to be grasped, a plurality of sensor data associated with the plurality of flex sensors mounted in a plurality of fingers of a reconfigurable robotic gripper;
computing, by the one or more hardware processors, a position the object to be grasped based on the current sequence of images pertaining to the object to be grasped;
simultaneously computing, by the one or more hardware processors, a current magnitude of bending and a current configuration associated with each of the plurality of fingers based on the plurality of sensor data;
iteratively computing, by the one or more hardware processors, an optimal grasp control signal based on the position of the object to be grasped, the current configuration and the current magnitude of bending associated with the each of the at least three fingers using a trained reward based Reinforcement Learning (RL) model; and
grasping, by the reconfigurable robotic gripper, the object to be grasped based on the optimal grasp control signal, wherein gripper action is performed by actuating a motor associated to each of the plurality of fingers based on the optimal grasp control signal.

### Claim 3 (depends on 2)

The processor implemented method as claimed in claim 2, wherein the RL model is trained in a simulation environment and adapted to be used in a real-world environment using one or more domain adaptation and domain randomization techniques, and wherein the RL model is a reward based optimization model.

### Claim 4

One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:
receiving a data pertaining to an object grasping environment, wherein the data comprises a current sequence of images pertaining to an object to be grasped, a plurality of sensor data associated with the plurality of flex sensors mounted in a plurality of fingers of a reconfigurable robotic gripper;
computing a position the object to be grasped based on the current sequence of images pertaining to the object to be grasped;
simultaneously computing a current magnitude of bending and a current configuration associated with each of the plurality of fingers based on the plurality of sensor data;
iteratively computing an optimal grasp control signal based on the position of the object to be grasped, the current configuration and the current magnitude of bending associated with the each of the at least three fingers using a trained reward based Reinforcement Learning (RL) model; and
grasping the object to be grasped based on the optimal grasp control signal, wherein gripper action is performed by actuating a motor associated to each of the plurality of fingers based on the optimal grasp control signal.

### Claim 5 (depends on 4)

The one or more non-transitory machine-readable information storage mediums of claim 4, wherein the RL model is trained in a simulation environment and adapted to be used in a real-world environment using one or more domain adaptation and domain randomization techniques, and wherein the RL model is a reward based optimization model.