# US 12,002,177 B2 · Systems and methods for dynamic interaction with an augmented reality environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,177 B2
- Document ID: US12002177B2
- Status: Granted
- Kind: B2
- Application number: 18/395,020
- Filed: 2023-12-22
- Published: 2024-04-18
- Granted: 2024-06-04
- Earliest claimed: 2022-08-03
- Assignees: Eqpme Inc.
- Applicants: Eqpme Inc.
- Inventors: Alexander Mascarin; Stephan Peralta; Andrea Tuzi; Matthew David Presta; Michael James Presta
- Web page: https://patentgrep.com/patent/US12002177B2

## CPC

- G06F 1/1626: {with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- E02F 3/325: {Backhoes of the miniature type}
- E02F 9/205: {Remotely operated machines, e.g. unmanned vehicles (E02F3/8866 takes precedence)}
- E02F 9/261: {Surveying the work-site to be treated}
- G06F 1/1686: {the I/O peripheral being an integrated camera}
- G06F 1/1694: {the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06F 30/12: characterised by design entry means specially adapted for CAD, e.g. graphical user interfaces [GUI] specially adapted for CAD
- G06Q 10/0875: Itemisation or classification of parts, supplies or services, e.g. bill of materials
- G06Q 10/103: {Workflow collaboration or project management}
- G06Q 10/20: Administration of product repair or maintenance
- G06Q 30/015: Providing customer assistance, e.g. assisting a customer within a business location or via helpdesk
- G06Q 30/0251: Targeted advertisements
- G06Q 30/0277: {Online advertisement}
- G06Q 30/0621: {by configuring or customising goods or services}
- G06Q 30/0623: {by investigating goods or services}
- G06Q 30/0633: {Managing shopping lists, e.g. compiling or processing purchase lists (shipping orders G06Q10/083; order filling G06Q10/087)}
- G06Q 30/0641: {utilising user interfaces specially adapted for shopping}
- G06Q 30/0645: Rental transactions; Leasing transactions
- G06Q 50/08: Construction
- G06T 7/13: Edge detection
- G06F 2111/18: using virtual or augmented reality
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- E02F: DREDGING; SOIL-SHIFTING (winning peat E21C49/00)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Methods and systems are provided for dynamic interaction with an augmented reality environment. In some embodiments, the systems and methods are directed at dynamically interacting with machinery within the augmented reality environment via an augmented reality device. The method involves analyzing a site in which the machinery is to be operated to capture environment data; displaying on the augmented reality device an augmented reality representation of the machinery as an overlay of a portion of an augmented reality environment; receiving an interaction request for interacting with the augmented reality representation of the machinery; determining whether the interaction request can be completed in respect of the machinery within the site; and in response to determining that the interaction request can be completed, displaying the augmented reality representation of the machinery in accordance with the interaction request, otherwise, indicating that the interaction request cannot be completed within the site.

## Claims

### Claim 1

A method for dynamically interacting with an augmented reality environment via an augmented reality device, the method comprising:
capturing, with the augmented reality device, environment data for estimating a site boundary of a site in which an object is to be placed;
capturing, from a physical display, an object identifier identifying the object to be displayed within the augmented reality environment corresponding to the site;
identifying, from a data storage, an augmented reality representation associated with the object identifier;
displaying on the augmented reality device the augmented reality representation of the object as an overlay of a portion of the augmented reality environment corresponding to the site;
receiving an interaction request via the augmented reality device for interacting with the augmented reality representation of the object, the interaction request comprises an active engagement corresponding to one or more real-world usages of the object;
determining, based on the captured environmental data, whether the interaction request can be completed within the site; and
in response to determining that the interaction request can be completed, displaying, on the augmented reality device, the one or more augmented reality representations in accordance with the interaction request, otherwise, indicating on the augmented reality device that the interaction request cannot be completed within the site.

### Claim 2 (depends on 1)

The method of claim 1, wherein the physical display comprises the object.

### Claim 3 (depends on 1)

The method of claim 1, wherein receiving the interaction request comprises receiving the interaction request in respect of one or more components of the object.

### Claim 4 (depends on 1)

The method of claim 1 further comprises:
in response to determining the augmented reality representation associated with the object identifier is unavailable, evaluating one or more characteristics of the object; and
recommending a comparable object based on the one or more characteristics.

### Claim 5 (depends on 1)

The method of claim 1 further comprises receiving a transaction request for the object via the augmented reality device.

### Claim 6 (depends on 5)

The method of claim 5, wherein receiving the transaction request for the object comprises receiving the interaction request in respect of one or more components of the object.

### Claim 7 (depends on 1)

The method of claim 1 further comprises:
identifying, based on at least the object identifier and the augmented reality representation of the object, at least one of a compatible object and one or more compatible components for the object; and
displaying on the augmented reality device the at least one of the compatible object and the one or more compatible components for engagement via the augmented reality device.

### Claim 8 (depends on 1)

The method of claim 1, wherein the object identifier comprises a machinery identifier corresponding to a manufacturer identifier.

### Claim 9 (depends on 8)

The method of claim 8, wherein the interaction request comprises operating a component of the augmented reality representation of a machinery associated with the machinery identifier, the operation of the component of the augmented reality representation corresponding to a real-world operation of the component within the site.

### Claim 10 (depends on 9)

The method of claim 9, wherein indicating on the augmented reality device that the interaction request cannot be completed within the site comprises:
indicating the component of the machinery is unsuitable for the site; and
recommending an alternative component compatible for the machinery and suitable for the site.

### Claim 11 (depends on 9)

The method of claim 9 further comprises receiving a transaction request for the component of the machinery via the augmented reality device.

### Claim 12 (depends on 9)

The method of claim 9, wherein the interaction request comprises substituting the component on the machinery.

### Claim 13 (depends on 1)

The method of claim 1, wherein indicating on the augmented reality device that the interaction request cannot be completed within the site comprises recommending at least one alternative object suitable for the site.

### Claim 14 (depends on 13)

The method of claim 13, wherein recommending the at least one alternative object suitable for the site comprises determining the at least one alternative object based on one or more of a user preference and a historical user data.

### Claim 15 (depends on 1)

The method of claim 1, wherein the data storage comprises an augmented reality model library accessible by the augmented reality device via a network.

### Claim 16

A system for dynamically interacting with an augmented reality environment via an augmented reality device, the system comprising a processor operable to:
capture environment data for estimating a site boundary of a site in which a plurality of objects is to be placed; capture, from a physical display, an object identifier identifying the object to be displayed within the augmented reality environment corresponding to the site;
identify, from a data storage, an augmented reality representation associated with the object identifier;
display on the augmented reality device the augmented reality representation of the object as an overlay of a portion of the augmented reality environment corresponding to the site;
receive an interaction request via the augmented reality device for interacting with the augmented reality representation of the object, the interaction request comprises an active engagement corresponding to one or more real-world usages of the object;
determine, based on the captured environmental data, whether the interaction request can be completed within the site; and
in response to determining that the interaction request can be completed, display, on the augmented reality device, the one or more augmented reality representations in accordance with the interaction request, otherwise, indicating on the augmented reality device that the interaction request cannot be completed within the site.

### Claim 17 (depends on 16)

The system of claim 16, wherein the physical display comprises the object.

### Claim 18

The system of 16, wherein the processor is operable to receive the interaction request in respect of one or more components of the object.

### Claim 19 (depends on 16)

The system of claim 16, wherein the processor is operable to:
in response to determining the augmented reality representation associated with the object identifier is unavailable, evaluate one or more characteristics of the object; and
recommend a comparable object based on the one or more characteristics.

### Claim 20 (depends on 16)

The system of claim 16, wherein the processor is operable to receive a transaction request for the object via the augmented reality device.

### Claim 21 (depends on 20)

The system of claim 20, wherein the processor is operable to receive the interaction request in respect of one or more components of the object.

### Claim 22 (depends on 16)

The system of claim 16, wherein the processor is operable to:
identify, based on at least the object identifier and the augmented reality representation of the object, at least one of a compatible object and one or more compatible components for the object; and
display on the augmented reality device the at least one of the compatible object and the one or more compatible components for engagement via the augmented reality device.

### Claim 23 (depends on 16)

The system of claim 16, wherein the object identifier comprises a machinery identifier corresponding to a manufacturer identifier.

### Claim 24 (depends on 23)

The system of claim 23, wherein the interaction request comprises operating a component of the augmented reality representation of a machinery associated with the machinery identifier, the operation of the component of the augmented reality representation corresponding to a real-world operation of the component within the site.

### Claim 25 (depends on 24)

The system of claim 24, wherein the processor is operable to:
indicate the component of the machinery is unsuitable for the site; and
recommend an alternative component compatible for the machinery and suitable for the site.

### Claim 26 (depends on 24)

The system of claim 24, wherein the processor is operable to receive a transaction request for the component of the machinery via the augmented reality device.

### Claim 27 (depends on 24)

The system of claim 24, wherein the interaction request comprises substituting the component on the machinery.

### Claim 28 (depends on 16)

The system of claim 16, wherein the processor is operable to recommend at least one alternative object suitable for the site.

### Claim 29 (depends on 28)

The system of claim 28, wherein the processor is operable to determine the at least one alternative object based on one or more of a user preference and a historical user data.

### Claim 30 (depends on 16)

The system of claim 16, wherein the data storage comprises an augmented reality model library accessible by the augmented reality device via a network.