# US 10,933,526 B2 · Method and robotic system for manipulating instruments > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,933,526 B2 - Document ID: US10933526B2 - Status: Granted - Kind: B2 - Application number: 15/960,242 - Filed: 2018-04-23 - Published: 2019-10-24 - Granted: 2021-03-02 - Earliest claimed: 2018-04-23 - Assignees: General Electric Company - Applicants: General Electric Company - Inventors: Huan Tan; Weston Blaine Griffin; Lynn Ann DeRose; Yi Xu; Ying Mao; Balajee Kannan - Web page: https://patentgrep.com/patent/US10933526B2 ## CPC - B25J 9/161: {Hardware, e.g. neural networks, fuzzy logic, interfaces, processor} - B25J 9/1664: {characterised by motion, path, trajectory planning} - B25J 9/1679: {characterised by the tasks executed} - B25J 9/1697: {Vision controlled systems} - B25J 11/005: {Manipulators for mechanical processing tasks} - B25J 15/0608: {with magnetic holding means} - G05B 2219/39393: Camera detects projected image, compare with reference image, position end effector - G05B 2219/39567: Use electromagnetic attraction to bring robot hand in contact with workpiece - G05B 2219/40053: Pick 3-D object from pile of objects - G05B 2219/40564: Recognize shape, contour of object, extract position and orientation - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract An approach relates to manipulation of tools or instruments in the performance of a task by a robot. In accordance with this approach, sensor data is acquired and processed to identify a subset of instruments initially susceptible to manipulation. The instruments are then manipulated in the performance of the task based on the processed sensor data. ## Claims ### Claim 1 A processor-implemented method for manipulating one or more objects, comprising: acquiring sensor data related to a plurality of instruments in an environment in which a robot is located; processing the sensor data to identify one or more non-occluded instruments in the plurality of instruments, wherein processing the sensor data comprises: identifying one or more instruments of the plurality of instruments based on one or more radiofrequency identifiers present on the one or more instruments; estimating a pose for at least one of the one or more instruments based on identification of the one or more instruments; and inferring occlusion of the one or more instruments based on the pose to identify the one or more non-occluded instruments; and operating one or both of a manipulator and an effector of the robot to manipulate at least one of the one or more non-occluded instruments based on a task assigned to the robot. ### Claim 2 (depends on 1) The processor-implemented method of claim 1, wherein acquiring the sensor data comprises acquiring visual data of at least a portion of the environment in which the plurality of instruments is present. ### Claim 3 (depends on 1) The processor-implemented method of claim 1, wherein the plurality of instruments comprises tools used in the environment and wherein the task comprises sorting or moving the tools. ### Claim 4 (depends on 1) The processor-implemented method of claim 1, wherein identifying one or more additional instruments of the plurality of instruments comprises identifying one or more visual identifiers present on the one or more additional instruments. ### Claim 5 (depends on 1) The processor-implemented method of claim 1, wherein estimating the pose for the at least one of the one or more instruments comprises matching a template of each identified instrument with a corresponding pose of each identified instrument derived from the sensor data. ### Claim 6 (depends on 1) The processor-implemented method of claim 1, wherein inferring occlusion comprises generating one or more occupancy maps that model intersections between instruments. ### Claim 7 (depends on 1) The processor-implemented method of claim 1, wherein the steps of acquiring, processing and operating are repeated until all instruments of the plurality of instruments are manipulated in accordance with the task. ### Claim 8 (depends on 1) The processor-implemented method of claim 1, further comprising interacting with a database containing stored data to identify the one or more non-occluded instruments. ### Claim 9 (depends on 1) The processor-implemented method of claim 1, further comprising determining a grip point for a respective non-occluded instrument prior to manipulating the respective non-occluded instrument, wherein the effector grasps the respective non-occluded instrument by the grip point when manipulating the respective non-occluded instrument. ### Claim 10 A processor-implemented method for identifying non-occluded items comprising: acquiring an image representation of a plurality of instruments; identifying one or more instruments from among the plurality of instruments in the image representation based on one or more radiofrequency identifiers present on the one or more instruments; estimating a pose for at least one of the one or more instruments based on identification of the one or more instruments; and inferring occlusion of the one or more instruments based on the pose to identify one or more non-occluded instruments. ### Claim 11 (depends on 10) The processor-implemented method of claim 10, wherein identifying the one or more instruments comprises accessing a database to retrieve an identity for the one or more instruments using the one or more radiofrequency identifiers. ### Claim 12 (depends on 11) The processor-implemented method of claim 11, comprising identifying one or more visual identifiers comprising two-dimensional bar codes or three-dimensional bar codes on the one or more instruments. ### Claim 13 (depends on 10) The processor-implemented method of claim 10, wherein estimating the pose for the at least one of the one or more instruments comprises matching a stored template of each identified instrument with a corresponding pose of each identified instrument in the image representation. ### Claim 14 (depends on 10) The processor-implemented method of claim 10, wherein inferring occlusion comprises generating one or more occupancy maps that model intersections between instruments. ### Claim 15 A robotic manipulation system, comprising: a robot configured to manipulate instruments, the robot comprising: one or more sensors configured to generate sensor data of at least a portion of an environment in which the robot is located; a manipulator configured to be moved with respect to the instruments in performance of a task; and an effector positioned on the manipulator and configured to interact with the instruments in the performance of the task; and a processing component configured to: process the sensor data to identify one or more non-occluded instruments from among a plurality of instruments, wherein processing the sensor data comprises: identifying one or more instruments of the plurality of instruments based on one or more radiofrequency identifiers present on the one or more instruments; estimating a pose for at least one of the one or more instruments based on identification of the one or more instruments; and inferring occlusion of the one or more instruments based on the pose to identify the one or more non-occluded instruments; and operate one or both of the manipulator and the effector of the robot to manipulate at least one of the one or more non-occluded instruments based on the task. ### Claim 16 (depends on 15) The robotic manipulation system of claim 15, wherein identifying the one or more instruments comprises accessing a database to retrieve an identify for the one or more instruments using the one or more radiofrequency identifiers. ### Claim 17 (depends on 15) The robotic manipulation system of claim 15, wherein estimating the pose for the at least one of the one or more instruments comprises matching a template of each identified instrument with a corresponding pose of each identified instrument derived from the image data. ### Claim 18 (depends on 15) The robotic manipulation system of claim 15, wherein inferring occlusion comprises generating one or more occupancy maps that model intersections between instruments.