System and apparatus for augmented reality display and controls

An augmented reality system comprising:

• a vision output device for displaying virtual images, the virtual images including virtually displayed information and virtual controls associated with controllable devices;
• a sensing device capturing an image within a reference frame, the vision output device being captured within the image;
• and
• at least one processor in communication with the sensing device, the processor obtaining the captured image from the sensing device, the processor identifying the vision output device within the reference frame of the captured image, the processor localizing the vision output device within the reference frame of the captured image and identifying an absolute position and orientation of the vision output device within the reference frame of the captured image;
• wherein the vision output device generates virtual displays to a user utilizing the vision output device at respective locations based on the absolute position and orientation of the vision output device within reference frame of the captured image, wherein the sensing device captures a user's selection of a virtual control within the captured image, wherein the processor identifies the user's selection of the virtual control within the captured image and enables a control action of a controllable device associated with the selected virtual control.