# US 2026/0094384 A1 · Configuring industrial automation systems in extended reality environments

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0094384 A1
- Document ID: US20260094384A1
- Status: Published application
- Kind: A1
- Application number: 18/901,698
- Filed: 2024-09-30
- Published: 2026-04-02
- Earliest claimed: 2024-09-30
- Assignees: Rockwell Automation Technologies, Inc.
- Applicants: Rockwell Automation Technologies, Inc.
- Inventors: Thong T. Nguyen; Robert O. Bradford; Ryan C. Gariepy
- Web page: https://patentgrep.com/patent/US20260094384A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A system for configuring of an automation system through interactions with virtual objects in an extended-reality environment using a display device having an at least partially transparent display. Control or optimization rules are represented by virtual objects that can be moved to the affected location within an extended reality environment. Simulations can be performed subject to the rules before accepting the rules into the live automation system. Additional equipment, devices, or other assets can be added to a simulation. The additional assets are represented by virtual objects in the extend reality environment and appear to operate with the existing assets.

## Claims

### Claim 1

A system for affecting performance of an automation system through interactions with virtual objects in an extended-reality environment using a display device having an at least partially transparent display, the system comprising:
one or more memory devices having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
causing the display device to display a virtual object on the at least partially transparent display so that the virtual object appears in the extended-reality environment when viewed through the at least partially transparent display, the virtual object representing a rule for behavior of the automation system, the rule dependent on one or more attributes of the virtual object;
changing the one or more attributes of the virtual object based on gestures from a user interacting with the virtual object within the extended-reality environment; and
transmitting the rule represented by the virtual object to the automation system.

### Claim 2 (depends on 1)

The system of claim 1, wherein the one or more attributes comprise a location of the virtual object in the extended-reality environment, the operations further comprising changing the location responsive to a grab gesture and moving the virtual object to a new location in the extended-reality environment.

### Claim 3 (depends on 1)

The system of claim 1, wherein the one or more attributes comprise at least one of:
a location of the virtual object in the extended-reality environment;
a type of the rule; or
a value of a constraint.

### Claim 4 (depends on 1)

The system of claim 1, wherein the behavior of the automation system is affected by an optimization problem, wherein the rule represents a constraint for the optimization problem.

### Claim 5 (depends on 1)

The system of claim 1, wherein the automation system comprises at least one of:
an autonomous mobile robot;
independent cart technology; or
a motor or a motor drive.

### Claim 6 (depends on 1)

The system of claim 1, the operations further comprising performing a simulation of the automation system subject to the rule represented by the virtual object.

### Claim 7

One or more non-transitory computer-readable storage media having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to:
cause a display device to display a virtual object on an at least partially transparent display so that the virtual object appears in an extended-reality environment when viewed through the at least partially transparent display, the virtual object representing a rule for behavior of an automation system, the rule dependent on one or more attributes of the virtual object;
change the one or more attributes of the virtual object based on gestures from a user interacting with the virtual object within the extended-reality environment; and
transmit the rule represented by the virtual object to the automation system.

### Claim 8 (depends on 7)

The computer-readable storage media of claim 7, wherein the one or more attributes comprise a location of the virtual object in the extended-reality environment, wherein the location is changed responsive to a grab gesture and moving the virtual object to a new location in the extended-reality environment.

### Claim 9 (depends on 7)

The computer-readable storage media of claim 7, wherein the one or more attributes comprise at least one of:
a location of the virtual object in the extended-reality environment;
a type of the rule; or
a value of a constraint.

### Claim 10 (depends on 7)

The computer-readable storage media of claim 7, wherein the behavior of the automation system is affected by an optimization problem, wherein the rule represents a constraint for the optimization problem.

### Claim 11 (depends on 7)

The computer-readable storage media of claim 7, wherein the automation system comprises at least one of:
an autonomous mobile robot;
independent cart technology; or
a motor or a motor drive.

### Claim 12 (depends on 7)

The computer-readable storage media of claim 7, having additional instructions stored thereon that cause the one or more processors to perform a simulation of the automation system subject to the rule represented by the virtual object.

### Claim 13

A system for testing changes to an automation system in an extended-reality environment using a display device having an at least partially transparent display, the system comprising:
one or more memory devices having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
executing a simulation of the automation system, the simulation comprising (i) simulated existing assets representing real existing assets of the automation system and (ii) one or more simulated additional assets representing one or more real additional assets being considered as potential additions to the automation system;
causing the real existing assets of the automation system to operate in accordance with simulated operation of the simulated existing assets in the simulation;
causing the display device to display one or more virtual objects representing the one or more simulated additional assets on the at least partially transparent display so that the one or more virtual objects appear at one or more locations in the extended-reality environment defined by one or more locations of the one or more simulated additional assets in the simulation; and
updating one or more visual attributes of the one or more virtual objects in the extended-reality environment based on one or more time-varying attributes of the one or more simulated additional assets in the simulation.

### Claim 14 (depends on 13)

The system of claim 13, wherein the one or more real additional assets comprise at least one of:
an autonomous mobile robot;
a cart of independent cart technology;
a track of independent cart technology; or
a motor or a motor drive.

### Claim 15 (depends on 13)

The system of claim 13, wherein the one or more visual attributes comprise at least one of:
the one or more locations of the one or more simulated additional assets;
color of the one or more virtual objects; or
brightness of the one or more virtual objects.

### Claim 16 (depends on 13)

The system of claim 13, wherein the one or more virtual objects representing the one or more simulated additional assets appear at the one or more locations when viewed from a changing viewer perspective through the at least partially transparent display.

### Claim 17 (depends on 13)

The system of claim 13, wherein the one or more real additional assets comprise objects carried by the automation system.

### Claim 18 (depends on 13)

The system of claim 13, wherein the one or more visual attributes of the one or more virtual objects is determined by a scale, the scale mapping a quantity of a simulated asset of the simulated existing assets or of the one or more simulated additional assets to the one or more visual attributes.

### Claim 19 (depends on 18)

The system of claim 18, wherein the one or more visual attributes is a color, a color saturation, or a brightness of the one or more virtual objects.

### Claim 20 (depends on 13)

The system of claim 13, wherein the automation system is subject to rules governing the simulated existing assets, wherein the rules are represented by rule virtual objects in the extended-reality environment.
