# US 2023/0056652 A1 · Robotic Grasping Via RF-Visual Sensing And Learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0056652 A1
- Document ID: US20230056652A1
- Status: Published application
- Kind: A1
- Application number: 17/819,685
- Filed: 2022-08-15
- Published: 2023-02-23
- Earliest claimed: 2021-08-13
- Assignees: Massachusetts Institute of Technology
- Applicants: Massachusetts Institute of Technology
- Inventors: Tara BOROUSHAKI; Isaac S. PERPER; Fadel ADIB
- Web page: https://patentgrep.com/patent/US20230056652A1

## CPC

- B25J 9/1612: {characterised by the hand, wrist, grip control}
- B25J 9/1697: {Vision controlled systems}
- B25J 13/006: {by means of a wireless system for controlling one or several manipulators}
- B25J 15/00: Gripping heads {and other end effectors (grippers used in machine tools B23Q7/04; gripping members fitted on cranes B66C1/42, B66C1/44; gripping means used for mounting electrical components H05K13/04; gripping means used in the manufacture of semiconductors H10P72/7602)}
- B25J 19/023: {including video camera means}
- B25J 19/027: {Electromagnetic sensing devices}
- H01Q 1/2208: {associated with components used in interrogation type services, i.e. in systems for information exchange between an interrogator/reader and a tag/transponder, e.g. in Radio Frequency Identification [RFID] systems (methods or arrangements for sensing record carriers, e.g. for reading patterns G06K7/00; record carrier for use with machines and with at least a part designed to carry digital markings G06K19/00)}
- H01Q 1/2216: {used in interrogator/reader equipment}
- H01Q 1/273: {Adaptation for carrying or wearing by persons or animals}
- G05B 2219/40499: Reinforcement learning algorithm
- G05B 2219/40577: Multisensor object recognition
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- H01Q: ANTENNAS, i.e. RADIO AERIALS (radiators or antennas for microwave heating H05B6/72)
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

Described is the design, implementation, and evaluation of a robotic system configured to search for and retrieve RFID-tagged items in line-of-sight, non-line-of-sight, and fully-occluded settings. The robotic system comprises a robotic arm having a camera and antenna strapped around a portion thereof (e.g. a gripper) and a controller configured to receive information from the camera and (radio frequency) RF information via the antenna and configured to use the information provided thereto to implement a method that geometrically fuses at least RF and visual information. This technique reduces uncertainty about the location of a target object even when the object is fully occluded. Also described is a reinforcement-learning network that uses fused RF-visual information to efficiently localize, maneuver toward, and grasp a target object. The systems and techniques described herein find use in many applications including robotic retrieval tasks in complex environments such as warehouses, manufacturing plants, and smart homes.

## Claims

### Claim 1

A robotic system comprising:
a robotic member having an end effector;
an imaging device coupled to the robotic member proximate to or on the end effector, the imaging device configured to provide visual information;
a radio frequency (RF) system configured to collect and provide information, the RF system including an antenna coupled to the robotic member proximate to or on the end effector; and
a controller in communication with the robotic member, the imaging device and the RF system, the controller configured to geometrically fuse RF information provided thereto by the RF system and visual information provided thereto by the imaging system to reduce uncertainty about a location of a target object even when the object is fully occluded.

### Claim 2 (depends on 1)

The robotic system of claim 1 wherein the end effector comprises a gripper and wherein the imaging device is coupled to a portion of the gripper.

### Claim 3 (depends on 1)

The robotic system of claim 1 wherein the end effector comprises a gripper and wherein the antenna is coupled to a portion of the gripper.

### Claim 4

A reinforcement-learning network comprising:
means for receiving fused RF-visual information and for processing the fused RF-visual information to efficiently localize, maneuver toward, and grasp a target object.