# US 2022/0118621 A1 · Enhanced robot safety perception and integrity monitoring > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0118621 A1 - Document ID: US20220118621A1 - Status: Published application - Kind: A1 - Application number: 17/561,747 - Filed: 2021-12-24 - Published: 2022-04-21 - Earliest claimed: 2021-12-24 - Assignees: Intel Corporation - Applicants: Intel Corporation - Inventors: Michael PAULITSCH; Florian GEISSLER; Ralf GRAEFE; Tze Ming HAU; Neslihan KOSE CIHANGIR; Ying Wei LIEW; Fabian Oboril; Yang PENG; Rafael ROSALES; Kay-Ulrich SCHOLL; Norbert STOEFFLER; Say Chuan TAN; Wei Seng YEAP; Chien Chern YEW - Web page: https://patentgrep.com/patent/US20220118621A1 ## CPC - B25J 9/1676: {Avoiding collision or forbidden zones} - B25J 9/1666: {Avoiding collision or forbidden zones} - B25J 9/1692: {Calibration of manipulator} - B25J 9/1697: {Vision controlled systems} - B25J 13/085: {Force or torque sensors (B25J13/082, B25J13/084 take precedence)} - B25J 13/089: {Determining the position of the robot with reference to its environment} - G05B 2219/40507: Distributed planning, offline trajectory, online motion, avoid collision - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract Disclosed herein are systems, devices, and methods for improving the safety of a robot. The safety system may determine a safety envelope of a robot based on a planned movement of the robot and based on state information about a load carried by a robot. The state information may include a dynamic status of the load. The safety system may also determine a safety risk based on a detected object with respect to the safety envelope. The safety system may also generate a mitigating action to the planned movement if the safety risk exceeds a threshold value. ## Claims ### Claim 1 A device comprising a processor configured to: determine a safety envelope of a robot based on a planned movement of the robot and based on state information about a load carried by a robot, wherein the state information comprises a dynamic status of the load; determine a safety risk based on a detected object with respect to the safety envelope; and generate a mitigating action to the planned movement if the safety risk exceeds a threshold value. ### Claim 2 (depends on 1) The device of claim 1, wherein the dynamic status of the load comprises changes in at least one of a mass of the load, a shape of the load, a height of the load, a width of the load, a volume of the load, a mounting point of the load on the robot, or a distance of the load from a center of gravity of the robot. ### Claim 3 (depends on 1) The device of claim 1, wherein the planned movement of the robot comprises at least one of a movement of the robot along a planned trajectory, a velocity of the movement of the robot along the planned trajectory, or a position of the robot along the planned trajectory. ### Claim 4 (depends on 1) The device of claim 1, wherein the processor is configured to determine the safety envelope based on at least one of a threshold breaking distance of the robot with the load, a threshold turn radius of the robot with the load, a velocity of the planned movement of the robot with the load, a trajectory of the planned movement of the robot with the load, or an acceleration of the planned movement of the robot with the load. ### Claim 5 (depends on 1) The device of claim 1, wherein the processor is configured to determine a predicted trajectory of the detected object based on at least one of past trajectories of other objects similar to the detected object, a velocity of the detected object, an acceleration of the detected object, a type of detected object, or a pose of the detected object. ### Claim 6 (depends on 5) The device of claim 5, wherein the processor is configured to receive the past trajectories from a machine learning model associated with the other objects. ### Claim 7 (depends on 1) The device of claim 1, wherein the processor is configured to receive the state information from a sensor configured to collect sensor data indicative of the state information. ### Claim 8 A device comprising a processor configured to: determine a deviation of a robot from a planned trajectory, wherein the deviation is based on a comparison of the planned trajectory with received sensor data indicative of an actual trajectory of the robot; determine a risk score associated with the deviation, wherein the risk score is based on identified objects within the actual trajectory; and generate a mitigation instruction if the risk score exceeds a threshold value, wherein the mitigation instruction comprises a revised trajectory for the robot; generate a takeover instruction if a difference between the revised trajectory and updated sensor data indicative of an updated actual trajectory of the robot exceeds a threshold difference. ### Claim 9 (depends on 8) The device of claim 8, the device further comprising a transmitter configured to transmit the mitigation instruction and the takeover instruction to the robot. ### Claim 10 (depends on 8) The device of claim 8, wherein the takeover instruction comprises an indication to activate to a safety subsystem of the robot. ### Claim 11 (depends on 9) The device of claim 9, wherein the transmitter is configured to transmit the mitigation instruction to the robot over a first communications channel, wherein the transmitter is configured to transmit the takeover instruction to the robot over a second communications channel, wherein the first communications channel is different from the second communications channel. ### Claim 12 (depends on 8) The device of claim 8, wherein the mitigation instruction comprises a transmission request for the robot to transmit diagnostic information to the device. ### Claim 13 (depends on 8) The device of claim 8, further comprising a memory configured to store at least one of the deviation, the risk score, the mitigation instruction, or the takeover instruction. ### Claim 14 A device comprising a processor configured to: determine a projected location of a robot at a measurement time based on received images of the robot; generate a diagnostic alarm if a reported positional location reported by the robot at the measurement time differs from the projected location by a threshold value. ### Claim 15 (depends on 14) The device of claim 14, further comprising a receiver, wherein the processor is configured to receive via the receiver the received images from a camera that is at a fixed location with respect to the robot. ### Claim 16 (depends on 14) The device of claim 14, wherein the projected location is defined with respect to a first coordinate system, wherein the processor configured to determine the projected location comprises the processor configured to convert an image position of the robot defined with respect to a second coordinate system into the projected location based on the fixed location of the camera, wherein the first coordinate system is different from the second coordinate system. ### Claim 17 (depends on 16) The device of claim 16, wherein the first coordinate system comprises a world coordinate system indicative of the robot within a world environment, wherein the second coordinate system comprises an image coordinate system indicative of the robot within the received images. ### Claim 18 (depends on 14) The device of claim 14, further comprising a memory configured to store at least one of the projected location, the measurement time, the received images, the diagnostic alarm, or the threshold value. ### Claim 19 (depends on 14) The device of claim 14, wherein the received images are associated with a first timeframe that is before the measurement time, wherein the processor is configured to determine the projected location based on an estimated trajectory of the robot in the first timeframe. ### Claim 20 (depends on 14) The device of claim 14, wherein the diagnostic alarm comprises a message indicating at least one of a request to perform a calibration, a request to perform a measurement, a perception malfunction associated with the received images, a motor malfunction of a motor on the robot, a motion control malfunction, or a communication malfunction of communications between the robot and the device. ### Claim 21 A device comprising: a means for determining a reliability level of a sensing means of a robot based on a difference between sensor data indicative of a current motion of the robot in an environment and expected sensor data for the environment and the current motion; a means for determining a risk assessment based on the reliability level of the sensing means and based on an expected movement of the robot; and a means for generating a mitigation plan for the robot if the risk assessment exceeds a threshold value. ### Claim 22 (depends on 21) The device of claim 21, wherein the mitigation plan comprises an instruction for the robot to calibrate the sensing means. ### Claim 23 (depends on 21) The device of claim 21, wherein the mitigation plan comprises an instruction for the robot to modify a parameter of the expected movement. ### Claim 24 (depends on 23) The device of claim 23, wherein the parameter of the expected movement comprises at least one of a speed, an acceleration, a trajectory, or a target location of the robot. ### Claim 25 (depends on 21) The device of claim 21, wherein the device further comprises a means for receiving the expected sensor data from a neural network model of trained sensor data.