Robotic learning of assembly tasks using augmented reality

A head-wearable device comprising:

• a display;
• a processor;
• and
• a memory storing instructions that, when executed by the processor, configure the head-wearable device to perform operations comprising:
• displaying a virtual object that corresponds to a physical object in a physical environment;
• tracking, using the head-wearable device, the virtual object by tracking hand gestures of a user of the head-wearable device that is co-located in the physical environment of the physical object;
• detecting, using the head-wearable device, the hands of the user grabbing the virtual object at a first location in the physical environment and releasing the virtual object at a second location in the physical environment;
• defining an initial state of the virtual object at the first location and a final state of the virtual object at the second location based on the detecting;
• and
• programming by demonstration a robotic system using the tracking of the virtual object, wherein the programming by demonstration comprises generating a robot program based on the initial state and the final state of the virtual object.

A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations comprising:

• displaying a virtual object in a display of a head-wearable device, the virtual object corresponding to a physical object in a physical environment;
• tracking, using the head-wearable device, the virtual object by tracking hand gestures of a user of the head-wearable device that is co-located in a physical environment of the physical object;
• detecting, using the head-wearable device, hands of the user grabbing the virtual object at a first location in the physical environment and releasing the virtual object at a second location in the physical environment;
• defining an initial state of the virtual object at the first location and a final state of the virtual object at the second location based on the detecting;
• and
• programming by demonstration a robotic system using the tracking of the virtual object, wherein the programming by demonstration comprises generating a robot program based on the initial state and the final state of the virtual object.