# US 10,769,853 B2 · Augmented reality interface and method of use > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,769,853 B2 - Document ID: US10769853B2 - Status: Granted - Kind: B2 - Application number: 14/585,018 - Filed: 2014-12-29 - Published: 2015-07-30 - Granted: 2020-09-08 - Earliest claimed: 2013-10-01 - Assignees: Myth Innovations, Inc. - Applicants: Myth Innovations, Inc. - Inventors: James Albert Luckett, Jr.; Chad Michael Rowlee; Shengli Fu - Web page: https://patentgrep.com/patent/US10769853B2 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G02B 27/017: {Head mounted} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/012: {Head tracking input arrangements} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry - G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) - G06F 15/82: data or demand driven - G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining - G09G 5/377: for mixing or overlaying two or more graphic patterns (G09G5/02, G09G5/397 take precedence) - H04L 63/0218: {Distributed architectures, e.g. distributed firewalls} - H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) - H04W 12/04: Key management, e.g. using generic bootstrapping architecture [GBA] - H04W 12/06: Authentication - H04W 12/068: {using credential vaults, e.g. password manager applications or one time password [OTP] applications} - H04W 12/08: Access security - H04W 12/088: using filters or firewalls - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/014: {comprising information/image processing systems} - G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye} - G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range - G06F 3/147: using display panels - G06F 2203/0384: Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices - H04L 63/0227: {Filtering policies (mail message filtering H04L51/212)} - H04W 4/02: Services making use of location information - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) ## Abstract A system for providing an augmented reality interface comprising a head mounted display having a wireless data interface, a camera and a display, the head mounted display configured to receive user interface data and to generate a user interface using the display and to receive image data from the camera and to transmit the image data over the wireless data interface to a wireless network. A wireless device configured to generate the wireless network, to transmit the user interface data to the head mounted display, to receive the image data from the head mounted display and to generate modified user interface data as a function of the image data. ## Claims ### Claim 1 A system for providing an augmented reality interface comprising: a first wireless device having a first processor under control of a first operating system and a second processor under control of a second operating system, the first wireless device configured to generate a wireless network, to receive image data of a plurality of items, to generate a virtual three dimensional coordinate system with an origin at a location of the first wireless device and to map a location of each of the plurality of items relative to the origin of the three dimensional coordinate system; a second wireless device remote from the first wireless device, the second wireless device having a third processor under control of the second operation system and an application, the second wireless device generating the image data; and wherein the first wireless device is further configured to control the application on the second wireless device using the second processor. ### Claim 2 (depends on 1) The system of claim 1 wherein the second wireless device is configured to generate augmented image data using the virtual three dimensional coordinate system generated by the first processor, the augmented image data comprising a plurality of objects, each having state and each having an associated coordinate relative to the origin of the three dimensional coordinate system. ### Claim 3 (depends on 1) The system of claim 1 further comprising a first virtual device remote from the first wireless device and the second wireless device, the first virtual device configured to receive object data and to generate object-based user interface data on a virtual display contained within a physical display. ### Claim 4 (depends on 3) The system of claim 3 wherein the first virtual device is relocatable between the first wireless device and the second wireless device and comprises a virtual display having an orientation that can be adjusted during relocation, wherein a user of a physical display can view the virtual display only from a predetermined direction relative to the virtual display. ### Claim 5 (depends on 1) The system of claim 1 wherein the first wireless device comprises a plurality of virtual machines, each associated with an image object that has state, each virtual machine having an associated address in the virtual three dimensional coordinate system, and wherein a user can use a hand motion to relocate each of the image objects within a physical display. ### Claim 6 (depends on 1) The system of claim 1 wherein the first wireless device comprises a plurality of virtual machines, each associated with an image object that has state, wherein the virtual machine for each image object is used to receive a user control that has been extracted from the image data for the image object by first identifying a location of a user's appendage within the image and by then mapping the location to the virtual three dimensional coordinate system as a function of a calibrated location associated with the user's appendage and wherein a user can use a hand motion to relocate each of the image objects within a physical display. ### Claim 7 (depends on 1) The system of claim 1 further comprising one or more user controls generated by the second wireless device and wherein a user can use a hand motion to relocate each of the image objects within a physical display. ### Claim 8 (depends on 1) The system of claim 1 wherein the first wireless device further comprises an augmented reality coordinate system configured to receive physical coordinate data for a location of the first wireless device and to generate virtual coordinate data for locations in a virtual field around the first wireless device, wherein the virtual coordinate data is different from the physical coordinate data by at least one field and wherein a user can use a hand motion to relocate an image object within a physical display, the image object having virtual coordinate data that is updated when the image object is relocated. ### Claim 9 (depends on 8) The system of claim 8 wherein the virtual coordinate data is incorporated into a unique Internet protocol address for a location within the virtual field and wherein a user can use a hand motion to relocate an image object within a virtual display that is contained within a physical display. ### Claim 10 (depends on 8) The system of claim 8 wherein the virtual coordinate data comprises three dimensional virtual coordinate data and a fourth field for identifying a virtual device and wherein a virtual display is associated with the virtual device and a user can use a virtual keyboard to enter data into the virtual display. ### Claim 11 (depends on 8) The system of claim 8 wherein the augmented reality coordinate system is configured to receive image data that includes real object image data and spatial grid image data and to generate virtual coordinate data for a real object associated with the real object image data as a function of location of the real object image data within the spatial grid image data and wherein the real object data can obscure a virtual display when the virtual coordinate data for the real object is between a user and virtual coordinate data for the virtual display. ### Claim 12 (depends on 8) The system of claim 8 wherein the augmented reality coordinate system is configured to receive wireless signal timing data associated with the real object and to modify the virtual coordinate data for the real object as a function of the wireless signal timing data and wherein a user can use a hand motion to relocate a first image object behind a second image object in a physical display, wherein the first image object has virtual coordinate data that is farther away from the user than virtual coordinate data of the second image object. ### Claim 13 A method for providing an augmented reality interface comprising: generating a user interface control in a user interface of a first wireless device, wherein the user interface is centered at a first virtual location relative to an origin having coordinates (0,0,0) located at a second wireless device; wirelessly receiving image data of a plurality of objects at the second wireless device from the first wireless device; processing the image data to extract a user motion and to generate a coordinate for each of the plurality of objects relative to the origin; and selecting a control function associated with the user interface control other than a control function to modify the user interface control as a function of each of the plurality of objects. ### Claim 14 (depends on 13) The method of claim 13 wherein the selected control function causes a second user interface control to be generated in the user interface of the first wireless device and only using image data, the second user interface control having a virtual location relative to the origin. ### Claim 15 (depends on 13) The method of claim 13 wherein the selected control function causes a second user interface control to be generated in the user interface of the first wireless device that accepts data entry from the user, the second user interface control comprising a keyboard and having a virtual location relative to the origin. ### Claim 16 (depends on 13) The method of claim 13 wherein the selected control function modifies a firewall setting of the user interface control, wherein the firewall setting inhibits access to the user interface control. ### Claim 17 (depends on 13) The method of claim 13 wherein the selected control function modifies a functional characteristic of the user interface control, wherein a virtual object having a virtual coordinate is relocated between the user interface control and the user and obscures the user interface control from the user. ### Claim 18 (depends on 13) The method of claim 13 wherein the selected control function modifies a functional characteristic of a second user interface control prior to generation of a second user interface control, wherein a virtual object having a virtual coordinate is relocated from a position between the user interface control and the user that obscures the user interface control from the user by a hand motion of the user. ### Claim 19 (depends on 13) The method of claim 13 wherein the selected control function modifies a functional characteristic of a second user interface control after generation of the second user interface control, wherein a virtual object having a virtual coordinate is relocated on top of the user interface control and the user from a position that obscures the user interface control from the user. ### Claim 20 A system for providing an augmented reality interface comprising: a first wireless device having means to receive image data of a plurality of objects, to generate a virtual three dimensional coordinate system with an origin at a location of the wireless device and to map the image data to the three dimensional coordinate system, to assign distinct coordinate data of the virtual three dimensional coordinate system to each of the plurality of objects; a second wireless device remote from the first wireless device, the second wireless device having means to generate the image data; and wherein the first wireless device is further configured to control the application on the second wireless device using the second processor and the coordinate data for each of the plurality of objects.