# US 11,443,495 B2 · Alignment- and orientation-based task assistance in an AR environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,443,495 B2
- Document ID: US11443495B2
- Status: Granted
- Kind: B2
- Application number: 16/237,285
- Filed: 2018-12-31
- Published: 2020-07-02
- Granted: 2022-09-13
- Earliest claimed: 2018-12-31
- Assignees: Palo Alto Research Center Incorporated
- Applicants: Palo Alto Research Center Incorporated
- Inventors: Hsuan-Ting Wu; Robert R. Price
- Web page: https://patentgrep.com/patent/US11443495B2

## CPC

- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06F 30/12: characterised by design entry means specially adapted for CAD, e.g. graphical user interfaces [GUI] specially adapted for CAD
- G06F 30/20: Design optimisation, verification or simulation (optimisation, verification or simulation of circuit designs G06F30/30)
- G06Q 10/06316: {Sequencing of tasks or work}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 2111/18: using virtual or augmented reality
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Embodiments described herein provide a system for facilitating dynamic assistance to a user in an augmented reality (AR) environment of an AR device. During operation, the system detects a first element of an object using an object detector, wherein the object is associated with a task and the first element is associated with a step of the task. The system then determines an orientation and an alignment of the first element in the physical world of the user, and an overlay for the first element. The overlay can distinctly highlight one or more regions of the first element and indicate how the first element fits in the object. The system then applies the overlay to the one or more regions of the first element at the determined orientation in the AR environment.

## Claims

### Claim 1

A method for facilitating dynamic assistance to a user in an augmented reality (AR environment of and AR device, comprising:
detecting, by the AR device, a first element of an object using an object detector, wherein the object is associated with a task;
determining that the first element is need for a step of the task;
projecting a hologram of the first element prior to locating the first element in an operating range of the AR device;
determining an orientation and an alignment of the first element in a physical world of the user;
determining an overlay for the first element, wherein the overlay distinctly highlights one or more regions of the first element, and wherein the overlay indicates how the first element fits in the object based on the one or more regions; and
highlighting the one or more regions in the AR environment by placing the overlay on the one or more regions of the first element at the determined orientation in the AR environment.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
determining one or more elements of the object that are attachable to the first element; and
projecting a second hologram depicting the first element attached to the one or more elements in the AR environment.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
enhancing a resolution of an image of the first element in the AR environment, wherein the enhancing including increasing, within the AR environment, prominence of a symmetry-breaking feature of the first element.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
determining a first region of the one or more regions of the first element, wherein the first region is attachable to a second region of a second element of the object; and
setting a same mark in the overlay for the first and second regions; and wherein placing the overlay comprises placing the same mark on the first and second regions.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
obtaining a three-dimensional model of the first element;
identifying the one or more regions in the three-dimensional model; and
placing the overlay based on the identified one or more regions in the three-dimensional model.

### Claim 6

An apparatus providing dynamic assistance to a user in an augmented reality (AR) environment, comprising:
a processor;
a storage device coupled to the processor and storing instructions that when executed by the processor cause the processor to perform a method, the method comprising:
detecting, by the AR device, a first element of an object using an object detector, wherein the object is associated with a task;
determining that the first element is needed for a step of the task;
projecting a hologram of the first element prior to locating the first element in an operating range of the apparatus;
determining an orientation and an alignment of the first element in physical world of the user;
determining an overlay for the first element, wherein the overlay distinctly highlights one or more regions of the first element, and wherein the overlay indicates how the first element fits in the object based on the one or more regions; and
highlighting the one or more regions in the AR environment by placing the overlay on the one or more regions of the first element at the determined orientation in the AR environment.

### Claim 7 (depends on 6)

The apparatus of claim 6, wherein the method further comprises:
determining one or more elements of the object that are attachable to the first element; and
projecting a second hologram depicting the first element attached to the one or more elements in the AR environment.

### Claim 8 (depends on 6)

The apparatus of claim 6, wherein the method further comprises:
enhancing a resolution of an image of the first element in the AR environment, wherein the enhancing includes increasing, within the AR environment, prominence of a symmetry-breaking feature of the first element.

### Claim 9 (depends on 6)

The apparatus of claim 6, wherein the method further comprises:
determining a first region of the one or more regions of the first element, wherein the first region is attachable to a second region of a second element of the object; and
setting a same mark in the overlay for the first and second regions; and
wherein placing the overlay comprises placing the same mark on the first and second regions.

### Claim 10 (depends on 6)

The apparatus of claim 6, wherein the method further comprises:
obtaining a three-dimensional model of the first element;
identifying the one or more regions in the three-dimensional model; and
placing the overlay based on the identified one or more regions in the three-dimensional model.