# US 2026/0094385 A1 · Replay of industrial automation system operations in extended reality environments

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0094385 A1
- Document ID: US20260094385A1
- Status: Published application
- Kind: A1
- Application number: 18/901,844
- Filed: 2024-09-30
- Published: 2026-04-02
- Earliest claimed: 2024-09-30
- Assignees: Rockwell Automation Technologies, Inc.
- Applicants: Rockwell Automation Technologies, Inc.
- Inventors: Yuhong Huang; Thong T. Nguyen; Jacek Jonda; Robert O. Bradford; Eric M. Gardner; Adam M. Shea; IMTHIAZ AHMED AEJAZ AHMED; Jonathan D. Hoffman; Jamie J. Cechvala; Haithem Mansouri; Ryan C. Gariepy
- Web page: https://patentgrep.com/patent/US20260094385A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 15/20: Perspective computation
- G06T 19/003: {Navigation within 3D models or images}
- G02B 27/0172: {characterised by optical features}
- G02B 2027/014: {comprising information/image processing systems}
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS

## Abstract

A system for the replay of automation system operations in an extended reality environment with virtual objects using a display device having an at least partially transparent display. The system stores historical operational data. Upon entering a playback mode, the system generates virtual objects at historical locations representing the devices of the automation system. During playback sampled historical locations are delivered to the display device and the virtual object location is updated so that the automation system appears in motion. Interpolation may be performed between sampled device locations so improve playback speed or decrease the data that is stored. Automation system operations can be replayed in a smaller volume to allow the operator a holistic three-dimensional view of operations.

## Claims

### Claim 1

A system for monitoring and troubleshooting an automation system in an extended-reality environment with virtual objects using a display device having an at least partially transparent display, the system comprising:
one or more memory devices having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
obtaining historical location data indicating historical locations for a portion of the automation system at a plurality of historical times;
causing the display device to display the virtual objects representing the portion of the automation system on the at least partially transparent display such that the virtual objects appear at locations in the extended-reality environment based on the historical location data for a first historical time when viewed from a changing viewer perspective through the at least partially transparent display; and
updating the locations at which the virtual objects appear in the extended-reality environment based on the historical location data for subsequent historical times such that the virtual objects appear to be moving between the historical locations in the extended-reality environment.

### Claim 2 (depends on 1)

The system of claim 1, wherein the locations are updated at a rate such that time appears to progress faster or slower than real time.

### Claim 3 (depends on 2)

The system of claim 2, the operations further comprising storing a time of an event of interest, wherein the locations are updated at the rate such that time appears to progress slower than real time for a time period proximate the time of the event.

### Claim 4 (depends on 1)

The system of claim 1, the operations further comprising storing a plurality of times of events of interest and receiving, from a viewer, a selected time of the plurality of times for which to provide the historical location data.

### Claim 5 (depends on 1)

The system of claim 1, the operations further comprising interpolating between stored samples of the historical location data to determine the locations at which the virtual objects appear in the extended-reality environment.

### Claim 6 (depends on 5)

The system of claim 5, wherein the interpolating comprises at least one of:
linear interpolation;
spline interpolation;
curve fitting; or
using a dynamic system model to obtain best estimates of the locations.

### Claim 7 (depends on 1)

The system of claim 1, wherein the extended-reality environment comprises real-world devices displayed at current locations and the virtual objects displayed at the historical locations, the operations further comprising indicating the real-world devices of the extended-reality environment that are not updated based on the historical location data.

### Claim 8 (depends on 1)

The system of claim 1, wherein the extended-reality environment comprises a fixed volume that appears to remain in a fixed location in the extended-reality environment when viewed from the changing viewer perspective through the at least partially transparent display, wherein the virtual objects appear to remain within the fixed volume of the extended-reality environment as the virtual objects move between the historical locations and are shown from a perspective of a virtual camera that moves between the historical locations along with the virtual objects.

### Claim 9 (depends on 1)

The system of claim 1, wherein the extended-reality environment comprises a fixed volume that appears to remain in a fixed location in the extended-reality environment when viewed from the changing viewer perspective through the at least partially transparent display, wherein the virtual objects are positioned within the fixed volume in the extended-reality environment and a location of the fixed volume differs from an actual location of the portion of the automation system represented by the virtual objects in the fixed volume.

### Claim 10

One or more non-transitory computer-readable storage media having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to:
obtain historical location data indicating historical locations for a portion of an automation system at a plurality of historical times;
cause a display device to display virtual objects representing the portion of the automation system on an at least partially transparent display such that the virtual objects appear at locations in an extended-reality environment based on the historical location data for a first historical time when viewed from a changing viewer perspective through the at least partially transparent display; and
update the locations at which the virtual objects appear in the extended-reality environment based on the historical location data for subsequent historical times such that the virtual objects appear to be moving between the historical locations in the extended-reality environment.

### Claim 11 (depends on 10)

The computer-readable storage media of claim 10, wherein the locations are updated at a rate such that time appears to progress faster or slower than real time.

### Claim 12 (depends on 11)

The computer-readable storage media of claim 11, the instructions causing the one or more processors to store a time of an event of interest, wherein the locations are updated at the rate such that time appears to progress slower than real time for a time period proximate the time of the event.

### Claim 13 (depends on 10)

The computer-readable storage media of claim 10, the instructions causing the one or more processors to store a plurality of times of events of interest and receiving, from a viewer, a selected time of the plurality of times for which to provide the historical location data.

### Claim 14 (depends on 10)

The computer-readable storage media of claim 10, the instructions causing the one or more processors to interpolate between stored samples of the historical location data to determine the locations at which the virtual objects appear in the extended-reality environment.

### Claim 15 (depends on 14)

The computer-readable storage media of claim 14, wherein the interpolating comprises at least one of:
linear interpolation;
spline interpolation;
curve fitting; or
a simulation of the portion of the automation system.

### Claim 16 (depends on 10)

The computer-readable storage media of claim 10, the instructions causing the one or more processors to indicate, on the at least partially transparent display, real-world devices of the extended-reality environment that are not updated based on the historical location data.

### Claim 17 (depends on 10)

The computer-readable storage media of claim 10, wherein the extended-reality environment comprises a fixed volume that appears to remain in a fixed location in the extended-reality environment when viewed from the changing viewer perspective through the at least partially transparent display, wherein the virtual objects appear to remain within the fixed volume of the extended-reality environment as the virtual objects move between the historical locations and are shown from a perspective of a virtual camera that moves between the historical locations along with the virtual objects.

### Claim 18

A system for monitoring and troubleshooting an automation system in a virtual-reality environment with virtual objects on a headset display device the system comprising:
one or more memory devices having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
causing the headset display device to display the virtual objects representing a portion of the automation system on the headset display device such that the virtual objects appear at locations in the virtual-reality environment based on historical location data;
interpolating between stored samples of the historical location data to determine the locations at which the virtual objects appear; and
updating the locations at which the virtual objects appear based on subsequent historical location data.

### Claim 19 (depends on 18)

The system of claim 18, wherein the locations are updated at a rate such that time appears to progress faster or slower than real time.

### Claim 20 (depends on 19)

The system of claim 19, the operations further comprising storing a time of an event of interest, wherein the locations are updated at the rate such that time appears to progress slower than real time for a time period proximate the time of the event.
