# US 2025/0144812 A1 · An augmented reality-based automation system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0144812 A1
- Document ID: US20250144812A1
- Status: Published application
- Kind: A1
- Application number: 18/837,023
- Filed: 2023-02-10
- Published: 2025-05-08
- Earliest claimed: 2022-02-14
- Assignees: BASF SE
- Applicants: BASF SE
- Inventors: Michael Hartmann; Hans Rudolph
- Web page: https://patentgrep.com/patent/US20250144812A1

## CPC

- B25J 13/00: Controls for manipulators (program controls B25J9/16)
- G05B 23/0216: {Human interface functionality, e.g. monitoring system providing help to the user in the selection of tests or in its configuration}
- B25J 9/1682: {Dual arm manipulator; Coordination of several manipulators}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path}
- G09G 3/001: {using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background (slide projectors per se G03B23/00 = 42 HP)}
- G05B 2219/23148: Helmet display, mounted on head of operator
- G05B 2219/24012: Use camera of handheld device, head mounted display
- G05B 2219/25062: Detect physical location of field device
- G05B 2219/32014: Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command
- G05B 2219/35482: Eyephone, head-mounted 2-D or 3-D display, also voice and other control
- G05B 2219/35487: Display and voice output incorporated in safety helmet of operator
- Y02P 90/02: Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- Y02P: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS

## Abstract

An automation system controls an automated production process of an industrial plant including a plurality of equipment for performing the production process. The system includes an augmented reality system, including a database system configured to store data and spatial coordinates associated with respective equipment of the plurality of equipment, augmented reality glasses configured to display the data, and executable program logic coupled to the augmented reality glasses. The executable program logic receives an up-to-date spatial position of a user from a positioning system and the data of equipment that are located in a proximity of the up-to-date spatial position of the user from the database system and control the augmented reality glasses for display of at least some of the received data.

## Claims

### Claim 1

An automation system for controlling an automated production process of an industrial plant, the industrial plant including a plurality of equipment for performing the production process, the plurality of equipment being spatially distributed in the industrial plant, the automation system comprising:
an augmented reality system, including:
a database system configured to store data and spatial coordinates associated with respective equipment of the plurality of equipment;
augmented reality glasses configured to display the data; and
executable program logic coupled to the augmented reality glasses, the executable program logic configured to receive an up-to-date spatial position of a user wearing the augmented reality glasses from a positioning system and the data of equipment that are located in a proximity of the up-to-date spatial position of the user from the database system, the executable program logic further configured to control the augmented reality glasses for display of at least some of the received data.

### Claim 2 (depends on 1)

The automation system of claim 1, wherein the augmented reality system further includes the positioning system, the positioning system configured for sensing a spatial position of the user in the plant.

### Claim 3 (depends on 2)

The automation system of claim 2, wherein the positioning system includes a spatial mapping mesh covering at least a portion of the plant, an indoor positioning system, a cellular positioning system, or a GPS receiver configured to determine the spatial position.

### Claim 4 (depends on 3)

The automation system of claim 3, wherein the spatial mapping mesh comprises spatial anchors, wherein the spatial anchors are interconnected, and wherein each spatial anchor relates a coordinate system associated with the augmented reality glasses for the display of the data to a spatial coordinate system associated with the spatial coordinates of one or more of the plurality of equipment stored in the database system, wherein the program logic is configured to display the data of equipment that is located in a proximity of the up-to-date spatial position of the user via the augmented reality glasses at predefined positions relative to the spatial anchors.

### Claim 5 (depends on 1)

The automation system of claim 1, wherein the augmented reality system is configured to create an avatar for at least one remote user, the remote user being remote to the industrial plant, wherein the program logic is configured to display the avatar in a proximity of the up-to-date spatial position of the user wearing the augmented reality glasses via the augmented reality glasses, wherein the augmented reality system is configured to provide a bidirectional visual and/or acoustic communication channel between the user wearing the augmented reality glasses and the remote user.

### Claim 6 (depends on 5)

The automation system of claim 5, wherein the augmented reality glasses comprise an acoustic output interface, wherein the augmented reality system is configured to provide a bidirectional acoustic communication channel between the user wearing the augmented reality glasses and the avatar, and wherein the program logic is configured to control the acoustic output interface such that the volume of the voice of the avatar output via the acoustic output interface to the user wearing the augmented reality glasses negatively correlates with distance of the user wearing the glasses and the avatar within the coordinate system associated with the augmented reality glasses for the display of the data.

### Claim 7 (depends on 5)

The automation system of claim 5,
wherein the automation system comprises a robot with a camera and/or a microphone, the robot being located in the industrial plant,
wherein the augmented reality system comprises a robot-control module configured to control movement and/or orientation of the robot such that a position and/or the orientation of the robot reflects a position and/or orientation of the avatar within the augmented reality displayed via the AR glasses, and
wherein the robot-control module is configured to automatically update the position and/or orientation of the avatar and the robot in accordance with a change in a position and/or orientation of the remote user or in response to a control command submitted by the remote user.

### Claim 8 (depends on 1)

The automation system of claim 1, the database system being a graph database system, a spatial database system and/or a streaming database system.

### Claim 9 (depends on 1)

The automation system of claim 1, further comprising:
a distributed control system coupled to the database system, the distributed control system, comprising:
a memory configured to store one or more process parameters in association with one or more of the equipment and to store process control software, and
a processor configured to execute the process control software for automatically controlling the production process.

### Claim 10 (depends on 9)

The automation system of claim 9, further comprising:
one or more sensors coupled to at least one equipment, the one or more sensors configured to measure the one or more process parameters of the at least one equipment, the one or more process parameters indicating d current state of the production process of the plant and/or a current state or mode of operation of the equipment; and
one or more actuators coupled to the at least one equipment, the one or more actuators configured to control the one or more process parameters or configured to be operated in accordance with one or more control parameters, wherein the distributed control system is coupled to the one or more sensors and the one or more actuators, wherein the processor is configured to execute the process control software for automatically controlling the production process based on measurement signals received from the one or more sensors and control signals sent to the one or more actuators.

### Claim 11 (depends on 1)

The automation system of claim 1, further comprising:
a user database coupled to a distributed control system, the user database configured to store user information associated with the production process, wherein the user information comprises user IDs, user roles and/or user privileges.

### Claim 12 (depends on 11)

The automation system of claim 11, wherein the executable program logic is configured to receive user credentials from the user and to authenticate the user, wherein a user ID of the user wearing the augmented reality glasses is associated with one or more of the user roles and each user role of the one or more user roles includes one or more of the user privileges, and wherein the executable program logic is further configured to select and display content of the at least some of the received data for display based upon the one or more user roles and/or the one or more user privileges associated with the user ID.

### Claim 13 (depends on 11)

The automation system of claim 11, wherein the one or more user roles includes one or more of a maintenance role and an engineering role, and the one or more user privileges defines access to one or more specific equipment of the plurality of equipment.

### Claim 14 (depends on 1)

The automation system of claim 1, wherein the data comprises one or more task instructions associated with the respective equipment.

### Claim 15 (depends on 1)

The automation system of claim 1, wherein one or more equipment of the plurality of equipment are one or more machines.

### Claim 16 (depends on 1)

The automation system of claim 1, wherein the executable program logic is configured to perform a query of the database based upon the up-to-date spatial position for retrieving, from the database, the data of equipment that are located in the proximity of the up-to-date spatial position of the user.

### Claim 17 (depends on 1)

The automation system of claim 1, wherein the database system is accessible via a subscription-based streaming service, and wherein the executable program logic is configured to receive from the database via the streaming service the data of equipment that are located in the proximity of the up-to-date spatial position of the user.

### Claim 18 (depends on 17)

The automation system of claim 17,
wherein the database system is configured to provide, in response to the query of the executable program logic, selectively the data of equipment that are located proximate to the up-to-date spatial position of the user wearing the augmented reality glasses, whereby a movement of the user relative to the equipment triggers the executable program logic to submit a new query to the database system comprising a new up-to-date spatial position of the user; or
wherein the database system is configured to provide, via the subscription-based streaming service, selectively the data of equipment that are located proximate to the up-to-date spatial position of the user wearing the augmented reality glasses, whereby a movement of the user relative to the equipment triggers the executable program logic to update the subscription with new up-to-date spatial position of the user.

### Claim 19 (depends on 1)

The automation system of claim 1, wherein the executable program logic is configured for:
receiving coordinates of visual objects graphically representing the data of the equipment proximate to the user wearing the augmented reality glasses;
determining said user's line of sight in the coordinate system of the real world;
receiving, from the database system, coordinates of real-world objects which are proximate to the up-to-date spatial position of the user wearing the augmented reality glasses;
determining if one or more of the virtual objects representing data of an equipment are positioned along the user's line of sight and are farther away from the user than at least one of the real world objects which is also positioned along the user's line of sight;
not displaying selectively the one or more virtual objects determined to be on the user's line of sight and be located farther away than at least one of the real-world objects.

### Claim 20 (depends on 1)

The automation system of claim 1, wherein the database system is a graph database system, the graph database system including a graph database having a plurality of interconnected nodes, wherein each node represents one or more objects of the industrial plant and includes spatial information associated with the one or more objects, and wherein the one or more objects are selected from the group consisting of a sensor, an actuator, raw material, and an equipment of the plurality of equipment, the equipment being e.g., a machine or a robot.

### Claim 21 (depends on 20)

The automation system of claim 20, wherein one or more of the nodes comprises or is stored in association with at least one respective container, each container being an isolated runtime environment instance comprising software for monitoring and/or operating at least one of the one or more objects represented by said node, and wherein the software in each container comprises a position reporting module configured to monitor a position of at least one of the one or more objects of the respective node and to update the spatial information associated with the at least one object of the respective node in the database system.

### Claim 22 (depends on 21)

The automation system of claim 21, wherein the spatial information associated with the at least one object of the respective node includes one or more spatial coordinates of the at least one object, the one or more spatial coordinates corresponding to one or more points on a bounding surface of the object and/or inside the bounding surface of the object.

### Claim 23 (depends on 1)

The automation system of claim 1, further comprising a network communicatively coupling the database system to the augmented reality glasses.

### Claim 24

A method of using an automation system for controlling an automated production process of an industrial plant, the industrial plant including a plurality of equipment for performing the production process, the plurality of equipment being spatially distributed in the industrial plant, the method comprising:
providing the automation system, the automation system comprising an augmented reality system, the augmented reality system including:
a database system having stored data and spatial coordinates associated with respective equipment of the plurality of equipment;
augmented reality glasses configured to display the data; and
executable program logic coupled to the augmented reality glasses,
receiving, by the executable program logic, an up-to-date spatial position of a user wearing the augmented reality glasses, from a positioning system;
determining, by the database system, the data of equipment that are located in a proximity of the up-to-date spatial position of the user;
receiving, by the executable program logic, the determined data of equipment from the database system;
controlling, by the executable program logic, the augmented reality glasses to display at least some of the received data, the displayed data enabling the user to control and/or maintain the automated production process.
