# US 10,431,007 B2 · Method and system for user interaction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,431,007 B2 - Document ID: US10431007B2 - Status: Granted - Kind: B2 - Application number: 15/609,113 - Filed: 2017-05-31 - Published: 2017-11-30 - Granted: 2019-10-01 - Earliest claimed: 2016-05-31 - Assignees: Augumenta Ltd. - Applicants: Augumenta Ltd. - Inventors: Damien Douxchamps; Tero Aaltonen; Peter Antoniac; Harri Kovalainen - Web page: https://patentgrep.com/patent/US10431007B2 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06F 1/163: {Wearable computers, e.g. on a belt} - G06F 1/1686: {the I/O peripheral being an integrated camera} - G06F 3/002: {Specific input/output arrangements not covered by G06F3/01 - G06F3/16 (other optical apparatus G02B27/00)} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/0425: {using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386)} - G06F 3/0426: {tracking fingers with respect to a virtual keyboard projected or printed on the surface (virtual keyboards on touch screens G06F3/04886)} - G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials - G06F 3/04886: by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract The present disclosure provides a method for user interaction. The method includes pointing a first device having a sensor, towards a panel defining a physical area having a plurality of markers rendered thereon, recognizing an interaction area defined by the plurality of markers on the panel, and rendering a plurality of graphical user interface elements on the panel in an Input/Output viewport. The plurality of markers includes at least one of an object and a shape recognizable by suitable computer vision means. Further, the Input/Output viewport includes a spatial volume between the first device and the panel. ## Claims ### Claim 1 A method for user interaction, the method comprising: pointing a sensor, mounted on head mounted smartglasses, towards a panel, wherein the smartglasses include an augmented reality display, and wherein the panel defines a physical area having a plurality of markers and a panel identifier; using the sensor to recognize the plurality of markers on the panel and an interaction area defined by the plurality of markers; using the sensor to recognize the panel identifier; using the recognized panel identifier to select a layout for a graphical user interface comprising a plurality of graphical user interface elements; and rendering the selected layout of the graphical user interface on the augmented reality display to appear to be floating in a spatial volume between the head mounted smartglasses and the interaction area defined by the plurality of markers; and to visually align with the panel; wherein at least one of the plurality of markers comprises at least one of information and a hyperlink to the information, wherein the information comprises at least one of a location, product information, and user information; wherein at least two of the plurality of markers are configured to be used as geometrical markers for defining a geometrical position and an orientation of the panel with respect to the head mounted smartglasses; and wherein the method further comprises: measuring a color of the interaction area and using the color measurement to affect the rendering of the identified plurality of graphical user interface elements; and randomizing positions of the identified plurality of graphical user interface elements when rendering the identified plurality of graphical user interface elements. ### Claim 2 (depends on 1) The method of claim 1, further comprising recognizing at least one of a hand, a finger and a predefined detectable object in the spatial volume between the head mounted smartglasses and the interaction area, wherein the at least one of the hand, the finger and the predefined detectable object is directed towards at least one of the plurality of graphical user interface elements as perceived by a user. ### Claim 3 (depends on 2) The method of claim 2, wherein the at least one of the hand, the finger and the predefined detectable object is directed for performing at least one control operation of at least one of the head mounted smartglasses and a second device. ### Claim 4 (depends on 1) The method of claim 1, further comprising rendering a new state of the plurality of graphical user interface elements in the spatial volume between the head mounted smartglasses and the interaction area. ### Claim 5 (depends on 4) The method of claim 4, wherein rendering the new state further comprises performing a perspective correction with one or more parameters defined by recognizing the plurality of markers and their geometrical relationship with each other. ### Claim 6 (depends on 1) The method of claim 1, further comprising combining a device identifier with the panel identifier for rendering a personalized content on the augmented reality display. ### Claim 7 (depends on 1) The method of claim 1, further comprising detecting movement of the head mounted smartglasses, and employing the detected movement to adjust a location of the interaction area. ### Claim 8 (depends on 1) The method of claim 1, wherein the plurality of markers and panel identifier are printed on the panel. ### Claim 9 A system for user interaction, comprising: a server comprising a database configured to store a plurality of device identifiers, a plurality of panel identifiers, and rendering information; head mounted smartglasses having a sensor, wherein the smartglasses include an augmented reality display, and wherein the sensor is configured to be used for pointing towards a panel defining a physical area having a plurality of markers and a panel identifier rendered thereon, wherein the head mounted smartglasses are configured to: recognize the plurality of markers on the panel and an interaction area defined by the plurality of markers; recognize the panel identifier; use the recognized panel identifier to select a layout for a graphical user interface comprising a plurality of graphical user interface elements; and render the selected layout of the graphical user interface on the augmented reality display to appear to be floating in a spatial volume between the head mounted smartglasses and the interaction area defined by the plurality of markers; and to visually align with the panel; wherein at least one of the plurality of markers comprises at least one of information and a hyperlink to the information, wherein the information comprises at least one of a location, product information, and user information, wherein at least two of the plurality of markers are configured to be used as geometrical markers for defining a geometrical position and an orientation of the panel with respect to the head mounted smartglasses; and wherein the system is configured to: measure a color of the interaction area and using the color measurement to affect the rendering of the identified plurality of graphical user interface elements; and randomize positions of the identified plurality of graphical user interface elements when rendering the identified plurality of graphical user interface elements. ### Claim 10 (depends on 9) The system of claim 9, wherein the head mounted smartglasses are further configured to recognize at least one of a hand, a finger and a predefined detectable object in the spatial volume between the head mounted smartglasses and the interaction area, wherein the at least one of the hand, the finger and the predefined detectable object is directed towards at least one of the plurality of graphical user interface elements. ### Claim 11 (depends on 9) The system of claim 9, wherein the head mounted smartglasses are further configured to render a new state of the plurality of graphical user interface elements in the spatial volume between the head mounted smartglasses and the interaction area. ### Claim 12 (depends on 11) system of claim 11, wherein the head mounted smartglasses are further configured to perform a perspective correction with one or more parameters defined by recognizing the plurality of markers and their geometrical relationship with each other when rendering the plurality of graphical user interface elements. ### Claim 13 (depends on 9) The system of claim 9, wherein the head mounted smartglasses are further configured to combine a device identifier with the panel identifier for rendering a personalized content on the augmented reality display. ### Claim 14 (depends on 9) The system of claim 9, wherein the head mounted smartglasses further comprise at least one motion sensor configured to detect movement of the head mounted smartglasses, and wherein the head mounted smartglasses employ the detected movement to adjust a location of the interaction area.