Virtual reality training simulator

A virtual reality system for training a user to assemble a device, the virtual reality system comprising:

• a display configured to display a three-dimensional (3D) training space, wherein the training space includes a work area having a work bench, an assortment of tools arranged in the work area, and components arranged in the work area;
• a hand sensor configured to detect a location of a hand of the user;
• memory configured to store:
• a collection of assemblages, wherein:
• each assemblage of the collection of assemblages is formed by combining parts of the assemblage;
• and
• the parts of the assemblage are a subset of the components arranged in the work area;
• and
• a collection of instructions, wherein:
• each assemblage of the collection of assemblages is associated with an instruction set from the collection of instructions;
• each instruction set of the collection of instructions includes an ordered list of steps performed to assemble the associated assemblage by combining the parts of the assemblage;
• and
• each step of the ordered list of steps:
• results in a sub-combination of the assemblage by at least one of adding at least one of the parts to the sub-combination, or processing at least one of the parts of the assemblage using at least one tool of the assortment of tools;
• and
• identifies at least one of a next part of the parts of the assemblage for adding to the sub-combination, a placement location for the next part, or a next tool of the assortment of tools for processing at least one of the parts of the assemblage;
• and
• processor circuitry configured to:
• generate and cause the display to display the training space;
• receive a selection of an assemblage from the collection of assemblages;
• receive from the collection of instructions an instruction set for the selected assemblage;
• and
• sequentially cause to be displayed on the display the ordered list of steps of the received instruction set including:
• displaying a current step of the ordered list of steps by:
• visually highlighting a next object in the training space, wherein the next object is the next part or the next tool;
• detecting with the hand sensor the user interacting with the highlighted next object based on a mapping to the training space of a measured location of the hand of the user;
• and
• upon the detecting of the user interacting with the highlighted next object, highlighting a target location either for placing the highlighted next object or for interacting with using the highlighted next object, wherein the target location includes a location in the work area, a tool of the assortment of tools, or a part of the assemblage;
• and
• when the user has completed the current step, repeatedly selecting the next step as the current step of the ordered list of steps and displaying the current step of the ordered list of steps until completing all of the steps in the instructions set resulting in the assembling of the associated assemblage,
• further comprising a haptic glove configured to provide haptic feedback to the hand of the user wearing the haptic glove based on a property of an object the user interacts with in the training space, wherein:
• the highlighted next object is a part of the assemblage;
• the highlighted target location is a highlighted portion of the highlighted next object;
• the user interacts with the highlighted portion by moving the hand of the user against the highlighted portion;
• and
• when the user interacts with the highlighted portion, the processor circuitry causes the haptic glove to provide haptic feedback to the user indicating a surface property of the highlighted portion,
• wherein:
• the surface property includes an existence of a burr on the highlighted portion;
• and
• when the user interacts with the highlighted portion and the highlighted portion includes the burr, the processor circuitry:
• causes the haptic glove to provide haptic feedback signifying the existence of the burr;
• highlights on the display as the next object a surface processing tool of the tools for removing the burr;
• and
• highlights on the display as the target location the highlighted portion including the burr.

An augmented reality system for assisting a user in the assembly of a device, wherein the augmented reality system comprises:

• a display configured to display a view of an environment of the user including a graphics overlay;
• processor circuitry configured to identify within a work area of the environment a location and a profile of:
• a work bench, an assortment of tools arranged in the work area, and components arranged in the work area;
• a hand sensor configured to detect a location of a hand of the user;
• memory configured to store:
• a collection of assemblages, wherein:
• each assemblage of the collection of assemblages is formed by combining parts of the assemblage;
• and
• the parts of the assemblage are a subset of the components arranged in the work area;
• a collection of instructions, wherein:
• each assemblage of the collection of assemblages is associated with an instruction set from the collection of instructions;
• each instruction set of the collection of instructions includes an ordered list of steps performed to assemble the associated assemblage by combining the parts of the assemblage;
• each step of the ordered list of steps:
• results in a sub-combination of the assemblage by at least one of adding at least one of the parts to the sub-combination, or processing at least one of the parts of the assemblage using at least one tool of the assortment of tools;
• identifies at least one of a next part of the parts of the assemblage for adding to the sub-combination, a placement location for the next part, or a next tool of the assortment of tools for processing at least one of the parts of the assemblage;
• wherein the processor circuitry is further configured to:
• map within the work area the location and the profile of the work bench, the assortment of tools, and the components;
• receive a selection of an assemblage from the collection of assemblages;
• receive from the collection of instructions an instruction set for the selected assemblage
• sequentially cause to be displayed on the graphics overlay the ordered list of steps of the received instruction set including:
• displaying a current step of the ordered list of steps by:
• visually highlighting a next object in the graphics overlay, wherein the next object is the next part or the next tool;
• detecting with the hand sensor the user interacting with the highlighted next object;
• and
• upon the detecting of the user interacting with the highlighted next object, highlighting in the graphics overlay a target location either for placing the highlighted next object or for interacting with using the highlighted next object, wherein the target location includes a location in the work area, a tool of the assortment of tools, or a part of the assemblage;
• and
• when the user has completed the current step, repeatedly selecting the next step as the current step of the ordered list of steps and displaying the current step of the ordered list of steps until completing all of the steps in the instructions set resulting in the assembling of the associated assemblage,
• further comprising a haptic glove configured to provide haptic feedback to the hand of the user wearing the haptic glove based on a property of an object the user interacts with in the training space, wherein:
• the highlighted next object is a part of the assemblage;
• the highlighted target location is a highlighted portion of the highlighted next object;
• the user interacts with the highlighted portion by moving the hand of the user against the highlighted portion;
• and
• when the user interacts with the highlighted portion, the processor circuitry causes the haptic glove to provide haptic feedback to the user indicating a surface property of the highlighted portion,
• wherein:
• the surface property includes an existence of a burr on the highlighted portion;
• and
• when the user interacts with the highlighted portion and the highlighted portion includes the burr, the processor circuitry:
• causes the haptic glove to provide haptic feedback signifying the existence of the burr;
• highlights on the display as the next object a surface processing tool of the tools for removing the burr;
• and
• highlights on the display as the target location the highlighted portion including the burr.