# US 11,618,166 B2 · Robot operating device, robot, and robot operating method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,618,166 B2 - Document ID: US11618166B2 - Status: Granted - Kind: B2 - Application number: 16/848,079 - Filed: 2020-04-14 - Published: 2020-11-19 - Granted: 2023-04-04 - Earliest claimed: 2019-05-14 - Assignees: Fanuc Corporation - Applicants: Fanuc Corporation - Inventors: Tingting Zhong; Shouta Takizawa - Web page: https://patentgrep.com/patent/US11618166B2 ## CPC - B25J 9/1689: {Teleoperation} - B25J 9/1602: {characterised by the control system, structure, architecture} - B25J 9/1664: {characterised by motion, path, trajectory planning} - B25J 9/1697: {Vision controlled systems} - B25J 13/06: Control stands, e.g. consoles, switchboards - B25J 13/081: {Touching devices, e.g. pressure-sensitive} - B25J 19/02: Sensing devices - B25J 19/023: {including video camera means} - G05B 2219/40003: Move end effector so that image center is shifted to desired position - G05B 2219/40392: Programming, visual robot programming language - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract A robot operating device includes a camera that is attached to a distal end of a robot arm or a position adjacent to the distal end and that acquires an image; a display which displays the image acquired by the camera; an operation-accepting unit which accepts an operation that is performed by an operator on the image displayed on the display unit; and a controller which moves the robot arm based on the operation accepted by the operation-accepting unit. ## Claims ### Claim 1 A robot operating device comprising: a camera that is attached to a distal end of a robot arm or a position adjacent to the distal end and that is configured to acquires an image; a touchscreen display that is configured to displays the image acquired by the camera and receives a tap operation or a swipe operation performed by an operator on the image displayed on the touchscreen display; and a controller which is configured to moves, just after the tap operation or the swipe operation, the robot arm based on a difference between a tapped position of the tap operation and a predetermined position in the image displayed on the touchscreen or based on swipe amount of a finger in the swipe operation received by the touchscreen display, wherein the touchscreen display is further configured to continue, during movement of the robot arm based on the tap operation or the swipe operation, to display the image which is acquired by the camera and which changes in response to the movement of the robot arm. ### Claim 2 (depends on 1) The robot operating device according to claim 1, wherein the touchscreen display is further configured to receives the tap operation for specifying an arbitrary position in the image, and the controller is further configured to moves the robot arm so as to move the arbitrary position towards a predetermined position in the image. ### Claim 3 (depends on 2) The robot operating device according to claim 2, wherein the controller is further configured to calculates a relationship between a stroke of the robot arm and a movement amount of an object in the image and, based on the calculated relationship, calculates the stroke of the robot arm necessary for moving the arbitrary position to the predetermined position. ### Claim 4 (depends on 2) The robot operating device according to claim 2, wherein the predetermined position in the image is a center of the image. ### Claim 5 (depends on 1) The robot operating device according to claim 1, wherein the touchscreen display is further configured to receives touch operation of a zoom command for the image, and the controller is further configured to moves the robot arm so as to move the camera in an optical axis direction of the camera based on the touch operation. ### Claim 6 (depends on 1) The robot operating device according to claim 1, wherein the touchscreen display is further configured to receives a touch operation of a rotation command on the image, and the controller is further configured to moves the robot arm so as to rotate the camera about an optical axis of the camera or about an axis parallel to the optical axis based on the touch operation. ### Claim 7 (depends on 1) The robot operating device according to claim 1, further comprising: a 9-axis sensor attached to the camera, wherein the controller is further configured to estimates an orientation of the camera in a coordinate system of the robot arm based on a detected value detected by the 9-axis sensor and controls the robot arm based on the orientation of the camera. ### Claim 8 A robot, comprising: a robot mechanical unit; a robot operating device comprising a camera that is attached to a distal end of a robot arm or a position adjacent to the distal end and that acquires an image; a touchscreen display that is configured to displays the image acquired by the camera and receives a tap operation or a swipe operation performed by an operator on the image displayed on the touchscreen display; and a controller which is configured to moves, just after the tap operation or the swipe operation, the robot arm based on a difference between a tapped position of the tap operation and a predetermined position in the image displayed on the touchscreen or based on swipe amount of a finger in the swipe operation received by the touchscreen display, wherein the touchscreen display is further configured to continue, during movement of the robot arm based on the tap operation or the swipe operation, to display the image which is acquired by the camera and which changes in response to the movement of the robot arm. ### Claim 9 A robot operating method comprising: acquiring an image with a camera attached to a distal end of a robot arm or a position adjacent to the distal end; displaying, on a touchscreen display, the image acquired by the camera; receiving a position of a touch operation performed by an operator on the displayed image or a swipe amount of a finger in a swipe operation performed by the operator on the displayed image; and moving, just after the tap operation or the swipe operation, the robot arm based on a difference between a tapped position of the tap operation and a predetermined position in the image displayed on the touchscreen or based on the swipe amount of the finger in the swipe operation wherein the touchscreen display continues, during movement of the robot arm based on the tap operation or the swipe operation, to display the image which is acquired by the camera and which changes in response to the movement of the robot arm. ### Claim 10 The robot operating device comprising: a camera that is attached to a distal end of a robot armor a position adjacent to the distal end and that acquires an image; a touchscreen display that is configured to display the image acquired by the camera and receive a tap operation or a swipe operation performed by an operator on the image displayed on the touchscreen display; and a controller which is configured to move the robot arm based on a difference between a tapped position of the tap operation and a predetermined position in the image displayed on the touchscreen or based on swipe amount of a finger in the swipe operation received by the touchscreen display, a deadman switch, wherein movement of the robot arm based on the tap operation or the swipe operation is performed only when the tap operation or the swipe operation is performed while the deadman switch is being pushed.