# US 2023/0141588 A1 · System and method for configuring augmented reality on a worksite

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0141588 A1
- Document ID: US20230141588A1
- Status: Published application
- Kind: A1
- Application number: 17/524,395
- Filed: 2021-11-11
- Published: 2023-05-11
- Earliest claimed: 2021-11-11
- Assignees: Caterpillar Paving Products Inc.
- Applicants: Caterpillar Paving Products Inc.
- Inventors: Brian D Nagel
- Web page: https://patentgrep.com/patent/US20230141588A1

## CPC

- G06Q 10/06316: {Sequencing of tasks or work}
- G05D 1/0044: {by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps (maps used for automatic navigation G05D1/0274; flight directors G01C23/005)}
- G06Q 10/063114: {Status monitoring or status determination for a person or group}
- G06Q 10/0633: Workflow analysis
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 20/20: in augmented reality scenes
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An augmented-reality device operating at a worksite obtains context data and worksite data relating to a user and a machine associated with the user. Based on a job role and a machine type identified for the user from the context data, a visual overlay for a mixed-reality display is selected from a plurality of visual overlays. The selected visual overlay is displayed for a scene in a window of the augmented-reality device in coordination with real-world images and status data, which is derived from the worksite data and also based on the job role and the machine type. While operating the machine, the user can view the selected visual overlay and status data, which may change for a new job role or machine.

## Claims

### Claim 1

A computer-implemented method, comprising:
receiving, by an electronic controller, an indication of activation of an augmented-reality device associated with a user at a worksite;
obtaining context data, by the electronic controller, relating to usage of the augmented-reality device at the worksite, the context data including a user identity for the user;
identifying, by the electronic controller, a first job role associated with the user identity within the worksite for the augmented-reality device;
generating, by the electronic controller, an augmented-reality overlay for the augmented-reality device specific to the user based at least in part on the first job role; and
causing, by the electronic controller, a first modification of a mixed-reality display of real-world images for a scene within a window of the augmented-reality device viewable by the user, the first modification including the augmented-reality overlay visually coordinated with the real-world images, wherein the first modification for the scene differs between the first job role and a second job role.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
receiving, by the electronic controller, worksite data relating to an activity of the user within the worksite;
filtering, by the electronic controller, the worksite data into job-specific data based at least in part on the first job role; and
causing, by the electronic controller, a second modification of the mixed reality display, the second modification including the job-specific data visually coordinated with the real-world images.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein the job-specific data comprises steps within a workflow executable by the user on the worksite within the scene.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the context data further includes a machine identity for a work machine operated by the user at the worksite, the augmented-reality overlay is specific to the first job role and the machine identity, and the first modification for the scene differs between the machine identity and another machine identity.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, further comprising:
obtaining updated context data, by the electronic controller, the updated context data including one of a new user identity for the user and a new machine identity for the work machine.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the electronic controller is integrated within the augmented-reality device.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the identifying the first job role comprises receiving an entry from the user via the augmented-reality device.

### Claim 8 (depends on 4)

The computer-implemented method of claim 4, wherein the first job role is an operator, the machine identity is a ground-shaping machine, and the augmented-reality overlay includes highlighting of a ground area to be shaped.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the first job role is an inspector and the augmented-reality overlay identifies locations within a ground area to be inspected.

### Claim 10

A computer-implemented method, comprising:
receiving, by an electronic controller, user data identifying a user of an augmented-reality device at a worksite;
identifying, by the electronic controller, a job role for the user at the worksite;
receiving, by the electronic controller, machine data identifying a work machine associated with the user at the worksite;
selecting, by the electronic controller, a visual overlay among a plurality of a visual overlays available for a scene viewable within the augmented-reality device, the visual overlay being selected among the plurality based at least in part on a combination of the job role and the work machine;
receiving, by the electronic controller, worksite data relating to operation of the work machine by the user at the worksite;
filtering, by the electronic controller, the worksite data into status data based at least in part on a combination of the job role and the work machine; and
causing, by the electronic controller, a modification of a mixed-reality display of real-world images for the scene within a window of the augmented-reality device viewable by the user, the modification including the visual overlay coordinated with the real-world images and the status data, wherein the modification for the scene is specific to the job role and the work machine.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10, further comprising:
receiving, by the electronic controller, worksite data relating to an activity of the user within the worksite;
filtering, by the electronic controller, the worksite data into job-specific data based at least in part on the job role; and
causing, by the electronic controller, an additional modification of the mixed reality display, the additional modification including the job-specific data visually coordinated with the real-world images.

### Claim 12 (depends on 10)

The computer-implemented method of claim 10, wherein the job role is one of supervisor, operator, and visitor.

### Claim 13 (depends on 10)

The computer-implemented method of claim 10, wherein the identifying the job role comprises receiving a job entry from the user via the augmented-reality device.

### Claim 14 (depends on 13)

The computer-implemented method of claim 13, further comprising, by the electronic controller, receiving a new job entry for the user via the augmented reality device, identifying a new job role for the user at the worksite, and selecting a new visual overlay among the plurality of visual overlays available for the scene based at least in part on the new job role.

### Claim 15

A system, comprising:
a work machine operable on a worksite by a user;
an augmented-reality device associated with the user; and
an electronic controller, coupled to at least the augmented-reality device, the electronic controller configured to:
receive a user identity for the user of the augmented-reality device at the worksite;
identify a first job role associated with the user identity within the worksite for the augmented-reality device;
generate an augmented-reality overlay for the augmented-reality device specific to the user based at least in part on the first job role; and
cause a modification of a mixed-reality display of real-world images for a scene within a window of the augmented-reality device viewable by the user, the modification including the augmented-reality overlay visually coordinated with the real-world images, wherein the modification for the scene differs between the first job role and a second job role.

### Claim 16 (depends on 15)

The system of claim 15, wherein the electronic controller is further configured to:
receive a machine identity for the work machine operated by the user,
wherein the augmented-reality overlay is generated specific to the user and the work machine based at least in part on the first job role and the machine identity.

### Claim 17 (depends on 16)

The system of claim 16, wherein the augmented-reality device establishes a wireless network with the work machine, and the work machine communicates the machine identity to the augmented-reality device over the wireless network.

### Claim 18 (depends on 16)

The system of claim 16, wherein the first job role is an operator, the machine identity is a ground-shaping machine, and the augmented-reality overlay includes a ground area to be shaped.

### Claim 19 (depends on 18)

The system of claim 18, wherein the electronic controller is further configured to receive a second job entry for the user via the augmented reality device, to identify the second job role for the user at the worksite, and to generate a new augmented-reality overlay for the scene based at least in part on the second job role.

### Claim 20 (depends on 19)

The system of claim 19, wherein the second job role is an inspector and the augmented-reality overlay identifies locations within a ground area to be inspected.
