Augmented reality (AR) system and method for tracking parts and visually cueing a user to identify and locate parts in a scene

An augmented reality (AR) system, comprising:

• a camera for recording images within a field-of-view (FOV) scanned over a larger field-of-regard (FOR) in a scene including parts;
• a display configured to display images of the scene in a FOV overlapping the camera FOV;
• one or more memory units;
• templates for the parts stored in said one or more memory units;
• a plurality of instructions stored in said one or more memory units, each said instruction including a list of parts and instructions for assembling or disassembling those parts;
• and
• a computer comprising:
• a classifier module that processes images of the scene using the templates to locate and identify the parts in the scene over the FOR, said classified parts' identities and locations relative to the FOR stored in said one or more memory units and retained in said memory units for parts that lie both inside the current FOV and outside the current FOV, said classifier module comprising a fiducial marker classifier that detects fiducial markers on parts and provides the part's identity and location and a parts classifier that uses the stored templates to detect parts and provides the part's identity and location;
• a pose module that processes data from images of the scene to estimate camera pose relative to the FOR, said pose module comprising at least two sources selected from
• the fiducial marker classifier, said classifier detecting fiducial markers on non-part scene features and providing the scene features identity and location in addition to the parts identity and locations;
• the parts classifier that provides identities and locations of parts;
• a visual scene feature classifier that identifies and locates points of interest in the scene;
• and
• an inertial sensor that estimates camera motion;
• and
• a camera pose estimator that processes inputs from at least two said sources and generates the estimated camera pose relative to the FOR;
• an instruction module that retrieves an instruction and a list of all parts relevant to the instruction;
• a tracker module that for each instruction determines from the list of all parts relevant to the instruction, locations of those parts relative to the FOR and the estimated camera pose relative to the FOR, a first list of parts and the locations of those parts that lie within the current FOV, a second list of parts that lie outside the current FOV and a third list of parts that have not been identified and located;
• a parts rendering module that for each instruction receives the first list of parts within the current FOV and a pose for each of those parts with respect to the camera and renders parts icons registered to parts within the current FOV, receives the second list of parts outside the current FOV, a pose for each of those parts with respect to the FOR and the estimated camera pose and renders parts icons with location indicators for parts outside the current FOV over the displayed images of the scene and receives the third list of parts and renders a list of those non-identified parts on the display;
• and
• an instruction-rendering module that renders the instructions for the use of those parts.

A method of augmented reality (AR), comprising:

• scanning a camera's field-of-view (FOV) over a larger field-of-regard (FOR) to record images of a scene including parts;
• processing the images to locate and identify parts in the scene relative to the FOR and to store and retain the parts' identities and locations for parts that lie both inside the current FOV and outside the current FOV;
• displaying an image of the scene in a FOV that overlaps the camera FOV;
• processing images of the scene from a current FOV to estimate a pose relative to the FOR;
• and
• for each of a sequence of instructions, each said instruction including a list of all parts and instructions for the use of those parts,
• using the locations of the parts relative to the FOR and the estimated pose relative to the FOR to determine a first list of the parts that lie within the current FOV and a second list of the parts that lie outside the current FOV from the list of parts associated with the execution of that instruction;
• using the first list of parts within the current FOV and a pose for each of those parts with respect the camera to render parts icons registered to the parts within the current FOV over the displayed image of the scene from the current FOV;
• using the second list of parts that lie outside the current FOV, a pose for each of those parts with respect the FOR and the estimated camera pose to render parts icons with location indicators for the parts outside the current FOV over the displayed image of the scene from the current FOV;
• and
• rendering the instructions for the use of those parts.