System and method for human interaction with virtual objects using reference device with fiducial pattern

A system for human interaction with virtual objects, comprising:

• a touch sensitive surface, configured to detect a position of a contact made on the touch sensitive surface;
• a reference layer rigidly attached underneath the touch sensitive surface and comprising:
• a fiducial layer providing one or more patterns comprised of light transmissive portions and light blocking portions;
• and
• a planar light source layer, spanning across the entirety of the light transmissive portions and the light blocking portions of the fiducial layer, and configured to cause the one or more patterns of the fiducial layer to be illuminated and at least semi-transparently displayed in the touch sensitive surface;
• and
• a wearable display device, configured to display a virtual object that is registered in a reference coordinate fixed with respect to the touch sensitive surface;
• one or more image sensors rigidly attached to the display device, configured to capture an image of at least a portion of the one or more patterns in the reference layer;
• and
• at least one processor, configured to:
• determine a position and an orientation of the display device with respect to the touch sensitive surface by analyzing image data about the one or more patterns found in the captured image;
• access touch data from the touch sensitive surface, the touch data comprising a detected position of contact made on the touch sensitive surface;
• cause the virtual object to be displayed by the wearable display device;
• generate a manipulation with the virtual object based on the detected position of the contact made on the touch sensitive surface;
• and
• cause the manipulation with the virtual object to be displayed by the wearable display device.