# US 2026/0245469 A1 · Multi-model augmented-reality instructions for assembly

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0245469 A1
- Document ID: US20260245469A1
- Status: Published application
- Kind: A1
- Application number: 19/489,517
- Filed: 2024-06-02
- Published: 2026-08-20
- Earliest claimed: 2023-06-02
- Assignees: The Texas A&M University System
- Applicants: The Texas A&M University System
- Inventors: Wei Yan; Seda Tuzun Canadinc
- Web page: https://patentgrep.com/patent/US20260245469A1

## CPC

- G09B 5/02: with visual presentation of the material to be studied, e.g. using film strip
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06N 20/00: Machine learning
- G06T 7/75: {involving models}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G09B 5/125: {the stations being mobile}
- G09B 19/003: {Repetitive work cycles; Sequence of movements}
- G09B 19/0069: {Engineering, e.g. mechanical, electrical design (computer engineering G09B19/0053)}
- G06T 2219/2008: Assembling, disassembling
- G06T 2219/2016: Rotation, translation, scaling
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 20/20: in augmented reality scenes
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Augmented-reality (AR) instructions for assembly of a physical object may be based on an assembly sequence divided into multiple assembly phases, using separate computational three-dimensional (3D) registration models (e.g., object recognition models) for the different phases. A user may be guided step by step through the assembly sequence, using a step counter to keep track of where in the assembly sequence the user is. In each step, an AR image may be created by rendering, overlaid onto the camera image, a virtual model of a part of the physical object to be added during the step, spatially registered with the physical object in its partially assembled state.

## Claims

### Claim 1

A method for providing augmented reality (AR) instructions for assembly of a physical object, the method being based on an assembly sequence divided into multiple assembly phases containing associated groups of steps, the method comprising:
providing, for each of the multiple assembly phases, an associated computational three-dimensional (3D) registration model; and
guiding a user step by step through at least a portion of the assembly sequence by performing operations comprising, for each step within the portion:
acquiring a camera image of the physical object in a partially assembled state;
using the 3D registration model associated with the assembly phase containing the step to determine a location and orientation of the physical object in the partially assembled state within the camera image;
creating an AR image by rendering, overlaid onto the camera image, a virtual model of a part of the physical object to be added during the step, spatially registered with the physical object in the partially assembled state; and
displaying the AR image on a display device.

### Claim 2 (depends on 1)

The method of claim 1, wherein the portion of the assembly sequence comprises a transition between two successive assembly phases, the method comprising switching the registration model from a registration model associated with a first one of the two successive assembly phase to a second one of the two successive assembly phases.

### Claim 3 (depends on 1)

The method of claim 1, maintaining a step counter that is incremented, for each step, responsive to input indicating completion of the step.

### Claim 4 (depends on 3)

The method of claim 3, wherein the input indication completion of the step is received from the user via a user input device.

### Claim 5 (depends on 4)

The method of claim 4, wherein the step counter is decremented in response to input, received at any step, indicating reversal of the step.

### Claim 6 (depends on 1)

The method of claim 1, wherein the registration models comprise one or more machine-learned object recognition models each associated with one of the multiple assembly phases and trained on image corresponding to perspective views of the physical object in a partially assembled state associated with one or more steps contained in the respective assembly phase, each image paired with a label indicating a ground-truth location and orientation of the physical object in the perspective view.

### Claim 7 (depends on 6)

The method of claim 6, wherein the images of the physical object in the partially assembled state are computed from a digital virtual model of the physical object in the partially assembled state.

### Claim 8 (depends on 7)

The method of claim 7, wherein computing the images from the digital virtual model comprises computationally rotating the digital virtual model to multiple different angles.

### Claim 9 (depends on 6)

The method of claim 6, wherein the registration models further comprise a machine-learned ground plane detection model associated with an initial assembly phase.

### Claim 10 (depends on 6)

The method of claim 6, wherein the multiple assembly phases are defined at least in part based on their impact on an accuracy of the associated registration models.

### Claim 11 (depends on 10)

The method of claim 10, wherein the multiple assembly phases are defined further based on their impact on at least one of user experience in the assembly of the physical object guided by the AR instructions, or structural stability of the physical object in partially assembled states during the assembly.

### Claim 12 (depends on 6)

The method of claim 6, wherein the one or more machine-learned object recognition models are trained on images of the physical object in a partially assembled state having asymmetric features.

### Claim 13 (depends on 1)

The method of claim 1, wherein creating the AR image further comprises rendering, overlaid onto the camera image, a wireframe model of the physical object in the partially assembled state.

### Claim 14 (depends on 1)

The method of claim 1, wherein creating the AR image further comprises rendering, overlaid onto the camera image, a transparent occlusion model of the physical model in the partially assembled state to represent occlusion of a virtual part by the physical object.

### Claim 15

A system to provide augmented-reality (AR) instructions for assembly of a physical object based on an assembly sequence divided into multiple assembly phases containing associated groups of steps, the system comprising:
a camera to acquire camera images of the physical object during the assembly;
a display device to display AR images;
one or more hardware processors; and
one or more machine-readable media storing:
for each of the multiple assembly phases, an associated computational three-dimensional (3D) registration model; and
machine-readable instructions which, when executed by the one or more hardware processors, cause the one or more hardware processor to perform operations for guiding a user step by step through at least a portion of the assembly sequence, the operations comprising, for each step within the portion:
acquiring a camera image of the physical object in a partially assembled state from the camera;
using the 3D registration model associated with the assembly phase containing the step to determine a location and orientation of the physical object in the partially assembled state within the camera image; and
creating an AR image for display on the display device by rendering, overlaid onto the camera image, a virtual model of a part of the physical object to be added during the step, spatially registered with the physical object in the partially assembled state.

### Claim 16 (depends on 15)

The system of claim 15, wherein the camera, display device, one or more hardware processors, and one or more machine-readable media are integrated into a single physical device.

### Claim 17 (depends on 16)

The system of claim 16, wherein the single physical device is a smartphone.

### Claim 18 (depends on 16)

The system of claim 16, wherein the single physical device is an AR headset.

### Claim 19 (depends on 15)

The system of claim 15, the operations further comprising maintaining a step counter that is incremented, for each step, responsive to input indicating completion of the step.

### Claim 20

A machine-readable medium storing machine-readable instructions which, when executed by one or more hardware processors, cause the one or more hardware processors to perform operations for providing augmented-reality (AR) instructions for assembly of a physical object based on an assembly sequence divided into multiple assembly phases containing associated groups of steps, the operations comprising:
providing, for each of the multiple assembly phases, an associated computational three-dimensional (3D) registration model; and
guiding a user step by step through at least a portion of the assembly sequence by performing operations comprising, for each step within the portion:
acquiring a camera image of the physical object in a partially assembled state;
using the 3D registration model associated with the assembly phase containing the step to determine a location and orientation of the physical object in the partially assembled state within the camera image;
creating an AR image by rendering, overlaid onto the camera image, a virtual model of a part of the physical object to be added during the step, spatially registered with the physical object in the partially assembled state; and
displaying the AR image on a display device.
