# US 12,759,155 B2 · Augmented reality supported safety plane adjustment > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,759,155 B2 - Document ID: US12759155B2 - Status: Granted - Kind: B2 - Application number: 18/726,577 - Filed: 2023-01-09 - Published: 2025-03-06 - Granted: 2026-10-13 - Earliest claimed: 2022-01-10 - Assignees: Universal Robots A/S - Applicants: Universal Robots A/S - Inventors: Krzysztof Zielinski; Jakob Schultz Ormhøj - Web page: https://patentgrep.com/patent/US12759155B2 ## CPC - B25J 9/1676: {Avoiding collision or forbidden zones} - G06T 11/00: Two-dimensional [2D] image generation - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract The invention relates to a method for AR assisted adjustment of a safety plane for a robot via a portable AR device comprising a physical display and a camera. The method comprises the steps of establishing a spatial reference system defined with reference to said robot. Connecting said portable AR device to a robot controller. Recording the surroundings of said portable AR device including said robot. Displaying said recording of said surroundings on said physical display. Establishing a virtual environment according to said spatial reference system overlaying said recorded surroundings, wherein said virtual environment comprises at least on safety plane. Manually perform, via said physical display, a safety plane displacement of said safety plane in said spatial reference system in said virtual environment, and transfer said displaced location of said safety plane to said robot controller. ## Claims ### Claim 1 A method for adjusting a location of a safety plane for a robot based on input from a portable augmented reality (AR) device comprising a display device and a camera, where the robot is controlled by a controller, and where the method comprises: establishing a spatial reference system with reference to the robot; establishing a communication connection between the portable AR device and the controller; obtaining, from the camera, a recording of surroundings of the portable AR device including the robot; causing the recording of the surroundings to be displayed on the display device; creating a virtual environment based on the spatial reference system, the virtual environment comprising a representation of the safety plane; causing the virtual environment to overlay the recording of the surroundings on the display device; and obtaining an adjusted location of the safety plane following displacement of the representation of the safety plane in the virtual environment in response to input from the display device, the input comprising a tap action and a drag action, and the displacement being a displacement that is along a normal vector to the safety plane. ### Claim 2 (depends on 1) The method of claim 1, wherein the spatial reference system comprises a three axis cartesian coordinate system having an origin at a base of the robot. ### Claim 3 (depends on 2) The method of claim 2, wherein the displacement of the representation of the safety plane comprises a displacement in one of the three axes in the three axis cartesian coordinate system. ### Claim 4 (depends on 1) The method of claim 1, wherein the displacement of the representation of the safety plane occurs when the robot is not in motion. ### Claim 5 (depends on 1) The method of claim 1, further comprising: establishing the safety plane using the AR device. ### Claim 6 (depends on 1) The method of claim 1, wherein the robot comprises a robotic arm; and wherein overlay of the representation of the safety plane on the display device is limited by an extent of a reach of the robotic arm. ### Claim 7 (depends on 1) The method of claim 1, wherein information about the safety plane is imported from the controller. ### Claim 8 (depends on 1) The method of claim 1, further comprising: receiving, at the controller and from the display device, data associated with the representation of the safety plane that has been manually approved. ### Claim 9 (depends on 1) The method of claim 1, wherein the AR device receives identifying information about the robot from the controller. ### Claim 10 (depends on 1) The method of claim 1, further comprising: automatically associating the representation of the safety plane with a real-world object in the recording. ### Claim 11 (depends on 1) The method of claim 1, further comprising: establishing a communication connection between an area scanner and the AR device, the area scanner monitoring at least part of the surroundings. ### Claim 12 A method for adjusting a location of a safety plane for a robot based on input from a portable augmented reality (AR) device comprising a display device and a camera, where the robot is controlled by a controller, and where the method comprises: establishing a spatial reference system with reference to the robot; establishing a communication connection between the portable AR device and the controller; obtaining, from the camera, a recording of surroundings of the portable AR device including the robot; causing the recording of the surroundings to be displayed on the display device; creating a virtual environment based on the spatial reference system, the virtual environment comprising a representation of the safety plane; causing the virtual environment to overlay the recording of the surroundings on the display device; obtaining, based on input from the display device, an adjusted location of the safety plane following displacement of the representation of the safety plane in the virtual environment; and undoing the displacement of the representation of the safety plane in the virtual environment. ### Claim 13 (depends on 12) The method of claim 12, wherein the displacement of the representation of the safety plane comprises a translation along a translation vector associated with the spatial reference system. ### Claim 14 (depends on 13) The method of claim 13, wherein the translation vector comprises a normal vector to the representation of the safety plane. ### Claim 15 (depends on 12) The method of claim 12, wherein the displacement of the representation of the safety plane comprises a displacement of an entirety of the representation of the safety plane in a direction that is normal to the representation of the safety plane. ### Claim 16 (depends on 12) The method of claim 12, wherein the displacement of the representation of the safety plane comprises a displacement of an entirety of the representation of the safety plane around one axis of a three axis cartesian coordinate system. ### Claim 17 (depends on 12) The method of claim 12, wherein the displacement of the representation of the safety plane comprises a displacement of an entirety of the representation of the safety plane relative to one point associated with the spatial reference system. ### Claim 18 (depends on 12) The method of claim 12, wherein the displacement of the representation of the safety plane is based on a tab-hold-slide gesture on the display device. ### Claim 19 A method for adjusting a location of a safety plane for a robot based on input from a portable augmented reality (AR) device comprising a display device and a camera, where the robot is controlled by a controller, and where the method comprises: establishing a spatial reference system with reference to the robot; establishing a communication connection between the portable AR device and the controller; obtaining, from the camera, a recording of surroundings of the portable AR device including the robot; causing the recording of the surroundings to be displayed on the display device; creating a virtual environment based on the spatial reference system, the virtual environment comprising a representation of the safety plane; causing the virtual environment to overlay the recording of the surroundings on the display device; and obtaining, based on input from the display device, an adjusted location of the safety plane following displacement of the representation of the safety plane in the virtual environment; wherein the displacement of the representation of the safety plane comprises a rotation around a rotation vector associated with the spatial reference system. ### Claim 20 (depends on 19) The method of claim 19, wherein the rotation vector comprises a plane vector lying on the representation of the safety plane. ### Claim 21 A method for adjusting a location of a safety plane for a robot based on input from a portable augmented reality (AR) device comprising a display device and a camera, where the robot is controlled by a controller, and where the method comprises: establishing a spatial reference system with reference to the robot; establishing a communication connection between the portable AR device and the controller; obtaining, from the camera, a recording of surroundings of the portable AR device including the robot; causing the recording of the surroundings to be displayed on the display device; creatin a virtual environment based on the spatial reference system, the virtual environment comprising a representation of the safety plane; causing the virtual environment to overlay the recording of the surroundings on the display device; and obtaining, based on input from the display device, an adjusted location of the safety plane following displacement of the representation of the safety plane in the virtual environment; wherein the safety plane is a first safety plane; and wherein the first safety plane is linked to a second safety plane so that movement of the first safety plane causes movement of the second safety plane. ### Claim 22 A system for adjusting a location of a safety plane for a robot, the system comprising: a controller configured to perform operations comprising: controlling the robot; and establishing a spatial reference system with reference to the robot; and a portable augmented reality (AR) device comprising a display device and a camera, the portable AR device being configured to perform operations comprising: capturing a recording of surroundings of the portable AR device including the robot; establishing a virtual environment based on the spatial reference system, the virtual environment comprising a representation of the safety plane; displaying the virtual environment on the display device, the virtual environment overlaying the recording, receiving an adjusted location of the safety plane based on a displacement of the representation of the safety plane in the virtual environment, the displacement being based on an input comprising a tap action and a drag action, and the displacement being a displacement that is along a normal vector to the safety plane; and providing the adjusted location to the controller. ### Claim 23 (depends on 22) The system of claim 22, wherein the spatial reference system comprises a three axis cartesian coordinate system having an origin at a base of the robot. ### Claim 24 (depends on 23) The system of claim 23, wherein the displacement of the representation of the safety plane comprises a displacement in one of the three axis of the three axis cartesian coordinate system. ### Claim 25 (depends on 22) The system of claim 22, wherein the representation of the safety plane is established on the portable AR device. ### Claim 26 (depends on 22) The system of claim 22, wherein the robot comprises a robotic arm; and wherein display of the representation of the safety plane on the display device is limited to an extent of a reach of the robotic arm. ### Claim 27 (depends on 22) The system of claim 22, wherein the representation of the safety plane is automatically associated with a real-world object in the recording.