US 12,403,590 B2 · Robotic grasping via RF-visual sensing and learning

Abstract

Described is the design, implementation, and evaluation of a robotic system configured to search for and retrieve RFID-tagged items in line-of-sight, non-line-of-sight, and fully-occluded settings. The robotic system comprises a robotic arm having a camera and antenna strapped around a portion thereof (e.g. a gripper) and a controller configured to receive information from the camera and (radio frequency) RF information via the antenna and configured to use the information provided thereto to implement a method that geometrically fuses at least RF and visual information. This technique reduces uncertainty about the location of a target object even when the object is fully occluded. Also described is a reinforcement-learning network that uses fused RF-visual information to efficiently localize, maneuver toward, and grasp a target object. The systems and techniques described herein find use in many applications including robotic retrieval tasks in complex environments such as warehouses, manufacturing plants, and smart homes.

Claims16

Dependent claims

A robotic system comprising:

A robotic member having an end effector;
An imaging device coupled to the robotic member proximate to or on the end effector, the imaging device configured to provide visual information;
A radio frequency (RF) system configured to collect and provide information, the RF system including an antenna coupled to the robotic member proximate to or on the end effector;
and
A controller in communication with the robotic member, the imaging device and the RF system, the controller configured to geometrically fuse RF distance information provided thereto by the RF system and visual information provided thereto by the imaging device to reduce uncertainty about a location of a target object even when the object is fully occluded,
Wherein the controller is further configured to:
Iteratively geometrically fuse the RF distance information and the visual information to reduce uncertainty about the target object location;
Determine whether the uncertainty is smaller than a threshold value;
and
In response to the uncertainty being smaller than the threshold value, cause the end effector to manipulate the target object.

7 more elements

An apparatus for locating a target object, the apparatus comprising:

An imaging device configured to collect visual data and provide visual information, the imaging device including a camera coupled to the apparatus;
A radio frequency (RF) system configured to collect and provide RF information, the RF system including an antenna coupled to the apparatus;
and
A processing module in communication with the imaging device and the RF system, the processing module configured to:
Estimate a range to the target object using the RF information, the range including one or more distance estimates between a location of the antenna and a location of the target object;
Generate an occupancy map of an environment of the imaging device using the visual information;
Geometrically fuse the range obtained from the RF information with the occupancy map from the visual information;
and
Use the fused information to identify one or more candidate regions for a location of the target object,
Wherein the imaging device and the RF system are mounted on a robotic end effector, and wherein the processing module is further configured to:
Iteratively geometrically fuse the range obtained from the RF information and the visual information to reduce the uncertainty about the target object location;
Determine whether the uncertainty is smaller than a threshold value;
and
In response to the uncertainty being smaller than the threshold value, cause the end effector to manipulate the target object.

11 more elements

Dependencies are inferred from the claim text.

Robotic grasping via RF-visual sensing and learning

Life of this patent

Robotic grasping via RF-visual sensing and learning

Life of this patent

Abstract

Claims16