# US 12,002,178 B2 · Systems and methods for dynamic interaction with an augmented reality environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,178 B2
- Document ID: US12002178B2
- Status: Granted
- Kind: B2
- Application number: 18/395,117
- Filed: 2023-12-22
- Published: 2024-04-18
- Granted: 2024-06-04
- Earliest claimed: 2022-08-03
- Assignees: Eqpme Inc.
- Applicants: Eqpme Inc.
- Inventors: Alexander Mascarin; Stephan Peralta; Andrea Tuzi; Matthew David Presta; Michael James Presta
- Web page: https://patentgrep.com/patent/US12002178B2

## CPC

- G06F 1/1626: {with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- E02F 3/325: {Backhoes of the miniature type}
- E02F 9/205: {Remotely operated machines, e.g. unmanned vehicles (E02F3/8866 takes precedence)}
- E02F 9/261: {Surveying the work-site to be treated}
- G06F 1/1686: {the I/O peripheral being an integrated camera}
- G06F 1/1694: {the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06F 30/12: characterised by design entry means specially adapted for CAD, e.g. graphical user interfaces [GUI] specially adapted for CAD
- G06Q 10/0875: Itemisation or classification of parts, supplies or services, e.g. bill of materials
- G06Q 10/103: {Workflow collaboration or project management}
- G06Q 10/20: Administration of product repair or maintenance
- G06Q 30/015: Providing customer assistance, e.g. assisting a customer within a business location or via helpdesk
- G06Q 30/0251: Targeted advertisements
- G06Q 30/0277: {Online advertisement}
- G06Q 30/0621: {by configuring or customising goods or services}
- G06Q 30/0623: {by investigating goods or services}
- G06Q 30/0633: {Managing shopping lists, e.g. compiling or processing purchase lists (shipping orders G06Q10/083; order filling G06Q10/087)}
- G06Q 30/0641: {utilising user interfaces specially adapted for shopping}
- G06Q 30/0645: Rental transactions; Leasing transactions
- G06Q 50/08: Construction
- G06T 7/13: Edge detection
- G06F 2111/18: using virtual or augmented reality
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- E02F: DREDGING; SOIL-SHIFTING (winning peat E21C49/00)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Methods and systems are provided for dynamic interaction with an augmented reality environment. In some embodiments, the systems and methods are directed at dynamically interacting with machinery within the augmented reality environment via an augmented reality device. The method involves analyzing a site in which the machinery is to be operated to capture environment data; displaying on the augmented reality device an augmented reality representation of the machinery as an overlay of a portion of an augmented reality environment; receiving an interaction request for interacting with the augmented reality representation of the machinery; determining whether the interaction request can be completed in respect of the machinery within the site; and in response to determining that the interaction request can be completed, displaying the augmented reality representation of the machinery in accordance with the interaction request, otherwise, indicating that the interaction request cannot be completed within the site.

## Claims

### Claim 1

A method for dynamically interacting with an augmented reality environment via an augmented reality device, the method comprising:
capturing, with the augmented reality device, environment data for estimating a site boundary of a site in which an object is to be placed;
displaying, on the augmented reality device, the augmented reality representation of the object as an overlay of a portion of the augmented reality environment corresponding to the site;
assigning at least one input prompt type for an interaction request to be received via the augmented reality device;
receiving the interaction request according to the at least one input prompt type via the augmented reality device for interacting with the augmented reality representation of the object, the interaction request comprises an active engagement corresponding to one or more real-world usages of the object; and
displaying, on the augmented reality device, the one or more augmented reality representations in accordance with the interaction request when the interaction request can be completed, otherwise, indicating on the augmented reality device that the interaction request cannot be completed within the site.

### Claim 2 (depends on 1)

The method of claim 1, wherein assigning the at least one input prompt type for the interaction request comprises switching between a first input prompt type to a second input prompt type different from the first input prompt type.

### Claim 3 (depends on 1)

The method of claim 1, wherein receiving the interaction request comprises receiving the interaction request in respect of one or more components of the object.

### Claim 4 (depends on 3)

The method of claim 3, wherein assigning the at least one input prompt type for the interaction request comprises:
assigning a first input prompt type for the interaction request in respect of the object; and
assigning a second input prompt type different from the first input prompt type for the interaction request in respect of at least one component of the one or more components of the object.

### Claim 5 (depends on 1)

The method of claim 1, wherein assigning the at least one input prompt type for the interaction request comprises:
determining the at least one input prompt type based on one or more of a user preference and a historical user data.

### Claim 6 (depends on 1)

The method of claim 1 further comprises receiving a transaction request for the object via the augmented reality device.

### Claim 7 (depends on 1)

The method of claim 1, wherein the object comprises a machinery and the interaction request comprises operating a component of the augmented reality representation of the machinery, the operation of the component of the augmented reality representation corresponding to a real-world operation of the component within the site.

### Claim 8 (depends on 7)

The method of claim 7, wherein indicating on the augmented reality device that the interaction request cannot be completed within the site comprises:
indicating the component of the machinery is unsuitable for the site; and
recommending an alternative component compatible for the machinery and suitable for the site.

### Claim 9 (depends on 8)

The method of claim 8 further comprises receiving a transaction request for the component of the machinery via the augmented reality device.

### Claim 10 (depends on 9)

The method of claim 9, wherein the interaction request comprises substituting the component on the machinery.

### Claim 11 (depends on 1)

The method of claim 1, wherein indicating on the augmented reality device that the interaction request cannot be completed within the site comprises recommending at least one alternative object suitable for the site.

### Claim 12 (depends on 1)

The method of claim 1, further comprises:
receiving a connection request via a network from a remote device to access the augmented reality device; and
wherein assigning the at least one input prompt type for the interaction request to be received via the augmented reality device comprises receiving the at least one input prompt type from the remote device.

### Claim 13 (depends on 12)

The method of claim 12 further comprises:
authenticating the connection request from the remote device to determine whether to grant the remote device access to the augmented reality device; and
in response to authenticating the connection request, mirroring a display of the remote device with a display of the augmented reality device.

### Claim 14 (depends on 13)

The method of claim 13, wherein receiving the interaction request according to the at least one input prompt type comprises receiving the interaction request from the remote device for interacting with the augmented reality representation of the object.

### Claim 15 (depends on 1)

The method of claim 1, wherein the at least one input prompt type for the interaction request comprises two or more input prompt types.

### Claim 16

A system for dynamically interacting with an augmented reality environment via an augmented reality device, the system comprising a processor operable to:
capture environment data for estimating a site boundary of a site in which a plurality of objects is to be placed;
display, on the augmented reality device, the augmented reality representation of the object as an overlay of a portion of the augmented reality environment corresponding to the site;
assign at least one input prompt type for an interaction request to be received via the augmented reality device;
receive the interaction request according to the at least one input prompt type via the augmented reality device for interacting with the augmented reality representation of the object, the interaction request comprises an active engagement corresponding to one or more real-world usages of the object; and
display, on the augmented reality device, the one or more augmented reality representations in accordance with the interaction request when the interaction request can be completed, otherwise, indicating on the augmented reality device that the interaction request cannot be completed within the site.

### Claim 17 (depends on 16)

The system of claim 16, wherein the processor is operable to switch between a first input prompt type to a second input prompt type different from the first input prompt type.

### Claim 18 (depends on 16)

The system of claim 16, wherein the processor is operable to receive the interaction request in respect of one or more components of the object.

### Claim 19 (depends on 18)

The system of claim 18, wherein the processor is operable to:
assign a first input prompt type for the interaction request in respect of the object; and
assign a second input prompt type different from the first input prompt type for the interaction request in respect of at least one component of the one or more components of the object.

### Claim 20 (depends on 16)

The system of claim 16, wherein the processor is operable to determine the at least one input prompt type based on one or more of a user preference and a historical user data.

### Claim 21 (depends on 16)

The system of claim 16, wherein the processor is operable to receive a transaction request for the object via the augmented reality device.

### Claim 22 (depends on 16)

The system of claim 16, wherein the object comprises a machinery and the interaction request comprises operating a component of the augmented reality representation of the machinery, the operation of the component of the augmented reality representation corresponding to a real-world operation of the component within the site.

### Claim 23 (depends on 22)

The system of claim 22, wherein the processor is operable to:
indicate the component of the machinery is unsuitable for the site; and
recommend an alternative component compatible for the machinery and suitable for the site.

### Claim 24 (depends on 23)

The system of claim 23, wherein the processor is operable to receive a transaction request for the component of the machinery via the augmented reality device.

### Claim 25 (depends on 24)

The system of claim 24, wherein the interaction request comprises substituting the component on the machinery.

### Claim 26 (depends on 16)

The system of claim 16, wherein the processor is operable to recommend at least one alternative object suitable for the site.

### Claim 27 (depends on 16)

The system of claim 16, wherein the processor is operable to:
receive a connection request via a network from a remote device to access the augmented reality device; and
receive the at least one input prompt type from the remote device.

### Claim 28 (depends on 27)

The system of claim 27, wherein the processor is operable to:
authenticate the connection request from the remote device to determine whether to grant the remote device access to the augmented reality device; and
in response to authenticating the connection request, mirror a display of the remote device with a display of the augmented reality device.

### Claim 29 (depends on 28)

The system of claim 28, wherein the processor is operable to receive the interaction request from the remote device for interacting with the augmented reality representation of the object.

### Claim 30 (depends on 16)

The system of claim 16, wherein the at least one input prompt type for the interaction request comprises two or more input prompt types.