# US 2012/0075343 A1 · Augmented reality (ar) system and method for tracking parts and visually cueing a user to identify and locate parts in a scene

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2012/0075343 A1
- Document ID: US20120075343A1
- Status: Published application
- Kind: A1
- Application number: 13/236,363
- Filed: 2011-09-19
- Published: 2012-03-29
- Earliest claimed: 2010-09-25
- Assignees: Teledyne Scientific & Imaging, LLC
- Inventors: STEVEN L. CHEN; Austin Eliazar
- Web page: https://patentgrep.com/patent/US20120075343A1

## CPC

- G09G 5/397: Arrangements specially adapted for transferring the contents of two or more bit-mapped memories to the screen simultaneously, e.g. for mixing or overlay (G09G5/02 takes precedence)
- G06T 7/73: using feature-based methods
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- A63F 2300/60: Methods for processing data by generating or executing the game program
- G06F 3/1431: {using a single graphics controller}
- G06T 2207/30164: Workpiece; Machine component
- G06T 2207/30244: Camera pose
- G09G 2340/125: wherein one of the images is motion video
- G09G 2370/16: Use of wireless transmission of display information
- G09G 2380/14: Electronic books and readers
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

An AR system both identifies and visually tracks parts for a user by maintaining spatial awareness of the user's pose and provides instructions to the user for the use of those parts. Tracking the identified parts, both inside and outside the current Field of View (FOV), and any missing parts for use with the current instruction improves the effectiveness and efficiency of both novice and experienced user alike.

## Claims

### Claim 1

An augmented reality (AR) system, comprising:
a camera for recording images within a field-of-view (FOV) over a larger field-of-regard (FOR) in a scene including parts;
a display;
one or more memory units;
templates for the parts stored in said one or more memory units;
a plurality of instructions stored in said one or more memory units, each said instruction including a list of parts and instructions for the use of those parts; and
a computer comprising:
a classifier module that processes images of the scene using the templates to locate and identify the parts in the scene over the FOR, said classified parts' identities and locations stored in said one or more memory units;
a pose module that processes data from images of the scene to estimate camera pose;
an instruction module that retrieves an instruction;
a tracker module that for each instruction determines from the estimated camera pose the list of parts and the locations of those parts that lie within the current FOV and those parts that lie outside the current FOV;
a scene-rendering module that displays the image of the scene within the current FOV on the display;
a parts rendering module that for each instruction renders parts icons registered to parts within the current FOV and renders parts icons with location indicators for parts outside the current FOV over the displayed images of the scene; and
an instruction-rendering module that renders the instructions for the use of those parts.

### Claim 2 (depends on 1)

The AR system of claim 1, wherein the parts comprise mechanical parts and tools.

### Claim 3 (depends on 2)

The AR system of claim 2, wherein the plurality of instructions comprises instructions to use the tools to assemble or disassemble the mechanical parts.

### Claim 4 (depends on 3)

The AR system of claim 3, wherein the plurality of instructions comprises instructions to use the tools to assemble or disassemble the mechanical parts onto or off of a vehicle.

### Claim 5 (depends on 4)

The AR system of claim 4, wherein the instructions rendering module renders the instruction on the display for the assembly or disassembly of the parts, said instruction including an illustration of the assembly or disassembly.

### Claim 6 (depends on 1)

The AR system of claim 1, wherein the camera FOV is scanned over the FOR to identify and locate all of the parts to perform all of the instructions and to store the identities and location of all of the parts, wherein the FOR is scanned and the parts stored prior to the use of any of the parts or the FOR is scanned as the parts icons and instructions are rendered as a user performs the instructions.

### Claim 7 (depends on 1)

The AR system of claim 1, wherein the classifier module comprises:
a fiducial marker classifier that detects fiducial markers on parts and uses the camera pose estimate to identify and locate the parts; and
a parts classifier that uses the stored templates to detect parts and uses the camera pose estimate to identify and locate the parts.

### Claim 8 (depends on 7)

The AR system of claim 7, wherein said pose module comprises:
at least two sources selected from
the fiducial marker classifier, said classifier detecting fiducial markers on non-part scene features and providing the scene features' identity and location in addition to the parts identity and locations;
the parts classifier that provides identities and locations of parts;
a visual scene feature classifier that identifies and locates points of interest in the scene; and
an inertial sensor that estimates camera motion; and
a camera pose estimator that processes inputs from at least two said sources and generates the estimated camera pose.

### Claim 9 (depends on 1)

The AR system of claim 1, wherein the tracker module that for each instruction determines any parts on the parts list that have not been identified and located, said parts rendering module rendering a list of non-identified parts on the display.

### Claim 10 (depends on 9)

The AR system of claim 9, wherein the tracker tracks the FOR for scanned and detected but not identified objects in the FOR, said tracker generating instructions for where to look for a missing part either within the scanned FOR at a detected but not identified object or outside the scanned FOR, said instructions rendering module rendering the instructions.

### Claim 11 (depends on 1)

The AR system of claim 1, wherein the instructions rendering module renders the instruction for the use of the parts on the display.

### Claim 12 (depends on 1)

The AR system of claim 1, wherein the system is a multi-user system comprising multiple cameras and displays for respective users, said multiple users work together to perform the same instruction with the same parts, said system rendering different parts icons depending on their respective FOV.

### Claim 13 (depends on 1)

The AR system of claim 1, wherein the system is a multi-user system comprising multiple cameras and displays for respective users, said multiple users work together to perform different instructions with different parts, said system rendering different parts icons depending on their respective parts lists and different FOV, said system rendering the different instructions to the different users.

### Claim 14

An augmented reality (AR) system for retrofitting a vehicle with a kit of parts including mechanical parts and tools, comprising:
a camera for recording images within a field-of-view (FOV) over a larger field-of-regard (FOR) in a scene including the vehicle and parts from the kit;
a display;
one or more memory units;
templates for the parts stored in said one or more memory units;
a plurality of instructions stored in said one or more memory units, each said instruction including a list of parts and instructions for assembling or disassembling those parts from the vehicle; and
a computer comprising:
a classifier module that processes images of the scene using the templates to locate and identify the parts in the scene over the FOR, said classified parts' identities and locations stored in said one or more memory units, said classifier module comprising a fiducial marker classifier that detects fiducial markers on parts and provides the part's identity and location and a parts classifier that uses the stored templates to detect parts and provides the part's identity and location;
a pose module that processes data from images of the scene to estimate camera pose, said pose module comprising at least two sources selected from
the fiducial marker classifier, said classifier detecting fiducial markers on non-part scene features and providing the scene features identity and location in addition to the parts identity and locations;
the parts classifier that provides identities and locations of parts;
a visual scene feature classifier that identifies and locates points of interest in the scene; and
an inertial sensor that estimates camera motion; and
a camera pose estimator that processes inputs from at least two said sources and generates the estimated camera pose;
an instruction module that retrieves an instruction;
a tracker module that for each instruction determines from the estimated camera pose the list of parts and the locations of those parts that lie within the current FOV and those parts that lie outside the current FOV;
a scene-rendering module that displays the image of the scene within the current FOV on the display;
a parts rendering module that for each instruction renders parts icons registered to parts within the current FOV and renders parts icons with location indicators for parts outside the current FOV over the displayed images of the scene; and
an instruction-rendering module that renders the instructions for the use of those parts on the display registered to the vehicle.

### Claim 15

A method of augmented reality (AR), comprising:
processing images of a scene from different Field-of-View (FOV) to locate and identify parts in the scene over a larger Field-of-Regard (FOR) and to store the parts' identities and locations;
processing images of the scene from a current FOV to estimate a pose; and
for each of a sequence of instructions, each said instruction including a list of parts and instructions for the use of those parts,
using the estimated pose to determine the parts that lie within the current FOV and the parts that lie outside the current FOV from the list of parts associated with the execution of that instruction;
rendering parts icons registered to the parts within the current FOV and rendering parts icons with location indicators for the parts outside the current FOV over a displayed image of the scene from the current FOV; and
rendering the instructions for the use of those parts.

### Claim 16 (depends on 15)

The method of claim 15, wherein the parts comprise mechanical parts and tools.

### Claim 17 (depends on 16)

The method of claim 16, wherein the plurality of instructions comprise instructions to use the tools to assemble or disassemble the mechanical parts onto or off of a vehicle.

### Claim 18 (depends on 15)

The method of claim 15, wherein processing the images from different FOV to locate and identify parts in the scene comprises processing the images with a fiducial marker classifier that detects fiducial markers on parts and uses the camera pose estimate to identify and locate the parts and a parts classifier that uses the stored templates to detect parts and uses the camera pose estimate to identify and locate the parts, and wherein processing images of the scene from the current FOV comprises processing the images with at least two sources selected from the fiducial marker classifier, said classifier detecting fiducial markers on non-part scene features and providing the scene features' identity and location in addition to the parts identity and locations, the parts classifier that provides identities and locations of parts, a visual scene feature classifier that identifies and locates points of interest in the scene and an inertial sensor that estimates camera motion, a camera pose estimator processing inputs from at least two said sources and generates the estimated camera pose.

### Claim 19 (depends on 15)

The method of claim 15, further comprising:
maintaining a list of located but not identified objects in the FOR, and
for each said instruction,
determining any parts on the parts list that have not been identified and located and rendering a list of non-identified parts on the display;
generating and rendering instructions for where to look for a missing part from among the list of located but not identified object in the FOR.

### Claim 20 (depends on 15)

The method of claim 15, wherein multiple users work together to perform the same instruction with the same parts, said method rendering different parts icons on the displayed image depending on the users' respective FOV or to perform different instructions with different parts, said method rendering different parts icons and instructions on the displayed image depending on users' respective parts lists and different FOV.
