# US 10,319,128 B2 · Augmented reality presentation of an industrial environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,319,128 B2
- Document ID: US10319128B2
- Status: Granted
- Kind: B2
- Application number: 15/391,213
- Filed: 2016-12-27
- Published: 2018-03-29
- Granted: 2019-06-11
- Earliest claimed: 2016-09-26
- Assignees: Rockwell Automation Technologies, Inc.
- Applicants: Rockwell Automation Technologies, Inc.
- Inventors: Sharon M. Billi-Duran; Christopher W. Como; Edward A. Gray; Kyle K. Reissner; Jonathan D. Walter; Mohit Singhai; Douglas J. Reichard; Scott N. Sandler; Ronald E. Bliss; Michael J. Pantaleano; Ryan Cahalane; Jessica L. Korpela; Bruce T. McCleave, Jr.
- Web page: https://patentgrep.com/patent/US10319128B2

## CPC

- G06Q 10/20: Administration of product repair or maintenance
- G05B 23/0205: {by means of a monitoring system capable of detecting and responding to faults}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 11/60: Creating or editing images; Combining images with text
- G09B 5/02: with visual presentation of the material to be studied, e.g. using film strip
- G05B 2219/32014: Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)

## Abstract

A industrial visualization system generates augmented reality presentations for rendering on a user's wearable computer as the user traverses an industrial facility. The system can leverage data indexed in a federated data model that unifies plant-wide data from multiple diverse sources under a common namespace. By leveraging the unified plant data and interacting with a wearable computer worn by a user, the augmented reality presentation system provides automation system data, notifications, and proactive guidance to a user via modification of the user's view of his or her immediate surroundings. Such modifications can include superimposing data values or indicators on the user's view of a machine or automation system through the user's wearable computer. The system can customize presentation of this information based on the user's role, location, line of sight, type of wearable device, and/or other contextual information.

## Claims

### Claim 1

A system for generation and delivery of augmented reality presentations, comprising:
a memory that stores executable components;
a processor, operatively coupled to the memory, that executes the executable components, the executable components comprising:
a device interface component configured to receive contextual data from a client device, the contextual data comprising at least location data identifying a location within an industrial environment and orientation data identifying an orientation of the client device;
a search component configured to, in response to a determination that the contextual data indicates that an industrial automation system is within a field of view of the client device, identify data items that are relevant to the industrial automation system, wherein a first copy of the data items are stored on an industrial device of the industrial automation system and a second copy of the data items are stored in a federated data model on the memory; and
a visualization component configured to generate an augmented reality presentation that renders the data items on the client device as respective overlays on the field of view,
wherein
the visualization component is further configured to:
while the client device is within a range of near field communication with the industrial device, select, as a source of the data items to be rendered on the augmented reality presentation, the first copy of the data items stored on the industrial device, retrieve the first copy of the data items, and render the first copy of the data items on the client device, and
while the client device is not within the range of near field communication with the industrial device, select, as the source of the data items to be rendered on the augmented reality presentation, the second copy of the data items stored in the federated data model, retrieve the second copy of the data items from the federated data model, and render the second copy of the data items on the client device, and
the device interface component is further configured to send the augmented reality presentation to the client device.

### Claim 2 (depends on 1)

The system of claim 1, wherein the client device is a wearable computer.

### Claim 3 (depends on 1)

The system of claim 1, the executable components further comprising:
a monitoring component configured to determine that one or more data items stored in the federated data model satisfy a defined criterion indicative of a performance issue relating to an industrial asset; and
a notification component configured to select a recipient for notification of the performance issue based on one or more notification rules, and to initiate delivery of a notification to a client device associated with the recipient via another augmented reality presentation by the visualization component.

### Claim 4 (depends on 3)

The system of claim 3, wherein the other augmented reality presentation comprises at least one of a directional icon that indicates a direction to a source of the performance issue relative to a current location of the client device associated with the recipient or a color-coded overlay positioned near a source of the performance issue.

### Claim 5 (depends on 3)

The system of claim 3, wherein
the device interface is further configured to receive environment data from the client device, and
the monitoring component is further configured to, in response to a determination that the environment data is indicative of a hazard or a performance problem of the industrial automation system, initiate delivery of another augmented reality presentation to the client device, the other augmented reality presentation rendering information about the hazard or performance issue.

### Claim 6 (depends on 5)

The system of claim 5, wherein the environment data comprises at least one of infrared data; heat signature data; vibration data; ambient noise level data; flux data; or data indicative of presence of a gas, particulates, smoke, or a toxin.

### Claim 7 (depends on 1)

The system of claim 1, wherein
the visualization component is configured to position a data item, of the data items, at a location within the augmented reality presentation that causes the data item to be positioned on or near a corresponding industrial device within the field of view of the client device, and
the location within the augmented reality presentation is determined based at least in part on the location data and the orientation data.

### Claim 8 (depends on 1)

The system of claim 1, wherein the visualization component is further configured to select the data items for presentation on the augmented reality presentation based in part on a user role associated with the client device.

### Claim 9 (depends on 1)

The system of claim 1, wherein the visualization component is configured to, in response to a determination that a data item, of the data items, has a value that is outside a defined range of values indicative of normal operation of the industrial automation system, render the data item on the augmented reality presentation in a manner that distinguishes the data item from other data items rendered on the augmented reality presentation.

### Claim 10

A method for generating augmented reality presentations for rendering of industrial data, comprising:
receiving, by a system comprising at least one processor, contextual data from a client device, wherein the contextual data comprises at least location data identifying a location of the client device within an industrial environment and orientation data identifying an orientation of the client device;
in response to determining that the contextual data signifies that an automation system is within a field of view of the client device:
identifying, by the system, data items that are relevant to the automation system,
selecting, by the system, a source of the data items from among a first copy of the data items stored on an industrial device of the automation system or a second copy of the data items stored on a federated data model maintained on a cloud platform, wherein the selecting comprises:
while the client device is within a range of a near field communication link with the industrial device, selecting, as the source of the data items, the first copy of the data items stored on the industrial device, and
while the client device is not within the range of the near field communication link with the industrial device, selecting, as the source of the data items, the second copy of the data items from the federated data model,
retrieving, by the system, the data items from the source of the data items,
generating, by the system, an augmented reality presentation that renders the data items on the client device as respective overlays on the field of view; and
sending, by the system, the augmented reality presentation to the client device.

### Claim 11 (depends on 10)

The method of claim 10, wherein the receiving the contextual data comprises receiving the contextual data from a wearable computer.

### Claim 12 (depends on 10)

The method of claim 10, further comprising:
determining, by the system, that one or more data items maintained in the federated data model satisfy a defined criterion indicative of a performance issue relating to an industrial asset; and
in response to the determining:
selecting, by the system, a recipient for notification of the performance issue based on one or more notification rules, and
sending, by the system, a notification to a client device associated with the recipient via another augmented reality presentation.

### Claim 13 (depends on 12)

The method of claim 12, wherein the sending the notification comprises rendering one or more directional icons on the other augmented reality presentation, the directional icons indicating a direction to a source of the performance issue relative to a current location of the recipient.

### Claim 14 (depends on 10)

The method of claim 10, wherein
the generating the augmented reality presentation comprises orienting a data item, of the data items, at a location on the augmented reality presentation that causes the data item to be positioned on or near a corresponding industrial device within the field of view of the client device, and
the location on the augmented reality presentation is determined based at least in part on the location data and the orientation data.

### Claim 15 (depends on 10)

The method of claim 10, wherein the searching comprises selecting at least one of the data items based at least in part on a user role associated with the client device.

### Claim 16 (depends on 10)

The method of claim 10, further comprising:
determining, by the system, that a data item, of the data items, associated with an industrial device has a value that is outside a defined tolerance indicative of normal operation of the industrial device; and
in response to the determining, rendering, by the system, the data item on the augmented reality presentation using at least one of a different color, a different size, or a different font relative to other data items rendered on the augmented reality presentation.

### Claim 17 (depends on 10)

The method of claim 10, further comprising:
monitoring, by the system, a speed of motion of the client device; and
in response to determining, based on the monitoring, that the speed exceeds a defined speed threshold, removing the data items from the field of view.

### Claim 18

A non-transitory computer-readable medium having stored thereon instructions that, in response to execution, cause a system comprising a processor to perform operations, the operations comprising:
receiving contextual data from a client device, the contextual data comprising at least location data that identifies a location of the client device within an industrial environment and orientation data identifying an orientation of the client device;
in response to determining that the contextual data indicates that an automation system is within a field of view of the client device, retrieving data items that are determined to be relevant to the automation system, wherein the retrieving comprises:
while the client device is within a range of near field communication with an industrial device that stores a first copy of the set of data items, retrieving, as the data items, the first copy of the data items from the industrial device, and
while the client device is outside the range of the near field communication with the industrial device, retrieving, as the data items, a second copy of the set of data items stored in a federated data model maintained on a cloud platform;
generating an augmented reality presentation that renders the data items on the client device as respective overlays on the field of view; and
sending the augmented reality presentation to the client device.

### Claim 19 (depends on 18)

The non-transitory computer-readable medium of claim 18, the operations further comprising:
in response to determining that a data item maintained in the federated data model has a value indicative of a performance issue relating to an industrial asset:
selecting a recipient for notification of the performance issue based on one or more notification rules; and
sending a notification to a client device associated with the recipient via another augmented reality presentation.

### Claim 20 (depends on 19)

The non-transitory computer-readable medium of claim 19, wherein the sending the notification comprises rendering one or more directional icons on the other augmented reality presentation, the one or more directional icons indicating a direction to a source of the performance issue relative to a current location of the client device associated with the recipient.