# US 2017/0355079 A1 · Robot system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2017/0355079 A1 - Document ID: US20170355079A1 - Status: Published application - Kind: A1 - Application number: 15/615,363 - Filed: 2017-06-06 - Published: 2017-12-14 - Earliest claimed: 2016-06-13 - Assignees: Fanuc Corporation - Applicants: Fanuc Corporation - Inventors: Yuuki TAKAHASHI - Web page: https://patentgrep.com/patent/US20170355079A1 ## CPC - B25J 9/1676: {Avoiding collision or forbidden zones} - B25J 9/1653: {parameters identification, estimation, stiffness, accuracy, error analysis} - B25J 9/1671: {characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems} - G06T 7/001: {using an image reference approach} - G06V 10/245: {by locating a pattern; Special marks for positioning} - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G05B 2219/34466: Bad circuits, watchdog, alarm, indication - G05B 2219/40202: Human robot coexistence - G05B 2219/40203: Detect position of operator, create non material barrier to protect operator - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract A robot system includes: a recognition means that recognizes an operator in distance image data as a security surveillance target; an area setting means that sets, in the distance image data, a common work area that a robot and the operator are allowed to enter; and a boundary setting means that sets, in the distance image data, a first boundary that the operator S is allowed to cross for entering the common work area but the robot is not allowed to cross, and a second boundary that the robot is allowed to cross for entering the common work area but the operator is not allowed to cross, wherein the crossing of the second boundary by the security surveillance target and the crossing of the first boundary by a non-target object not recognized as the security surveillance target are detected. ## Claims ### Claim 1 A robot system comprising: a distance image data acquisition device that chronologically and sequentially acquires distance image data containing at least a partial image of a work area for a robot and at least a partial image of an area that an operator or another robot is allowed to enter; a reference image data storage which stores at least one reference image data that is to be compared with an image appearing in the series of distance image data in order to recognize the operator or the other robot in the series of distance image data by the comparison; and a controller, wherein the controller conducts: a recognition process that refers to the reference image data and recognizes the operator or the other robot in the series of distance image data as a security surveillance target; an area setting process that sets a common work area in the series of distance image data, the common work area being an area that the robot is allowed to enter and the operator recognized as the security surveillance target or the other robot recognized as the security surveillance target is also allowed to enter; a boundary setting process that sets a first boundary and a second boundary in the series of distance image data, the first boundary being a boundary that the operator recognized as the security surveillance target or the other robot recognized as the security surveillance target is allowed to cross for entering the common work area but the robot or a non-target object not recognized as the security surveillance target is not allowed to cross, the second boundary being a boundary that the robot is allowed to cross for entering the common work area but the operator or the other robot is not allowed to cross; and a boundary crossing detection process that detects, in the series of distance image data, crossing of the second boundary by the security surveillance target and crossing of the first boundary by the non-target object not recognized as the security surveillance target. ### Claim 2 (depends on 1) The robot system according to claim 1, wherein the controller further conducts a robot stop process that stops the robot when the crossing by the security surveillance target or the non-target object is detected by the boundary crossing detection process. ### Claim 3 (depends on 1) The robot system according to claim 1, wherein, in the series of distance image data, an object which is integrated with the operator or the other robot and which moves together with the operator and the other robot is recognized as the security surveillance target in the recognition process. ### Claim 4 (depends on 2) The robot system according to claim 2, wherein the reference image data is compared to a mark provided on the operator or the other robot appearing in the series of distance image data, and is used to recognize, by the comparison, the object integrated with an image corresponding to the mark in the series of distance image data as the security surveillance target. ### Claim 5 (depends on 4) The robot system according to claim 4, wherein the robot stop process stops the robot when the image corresponding to the mark in the series of distance image data changes more greatly than a predetermined reference extent. ### Claim 6 (depends on 1) The robot system according to claim 1, further comprising a display device for displaying that the security surveillance target in the series of distance image data is being recognized by the recognition process. ### Claim 7 (depends on 1) The robot system according to claim 1, wherein the reference image data contains information regarding a height and a figure of the operator. ### Claim 8 (depends on 1) The robot system according to claim 1, wherein the reference image data storage stores robot reference image data that is to be compared with a robot mark provided on the robot in the series of distance image data in order to recognize the robot in the series of distance image data by the comparison, the robot reference image data are referred to and the robot in the series of distance image data as a security surveillance target robot is recognized by the recognition process, and crossing of the second boundary by an object other than the security surveillance target robot is detected in the series of distance image data by the boundary crossing detection process.