# US 10,466,858 B2 · Systems and methods for interacting with a virtual menu

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,466,858 B2
- Document ID: US10466858B2
- Status: Granted
- Kind: B2
- Application number: 14/556,622
- Filed: 2014-12-01
- Published: 2015-06-04
- Granted: 2019-11-05
- Earliest claimed: 2013-12-01
- Assignees: Upskill, Inc.
- Applicants: Brian Adams Ballard; James Leighton Athey; Jeffrey Edward Jenkins; Todd Richard Reily; Harold Ronald Villanueva Tagunicar; Michael Anthony Sciscenti
- Inventors: Brian Adams Ballard; James Leighton Athey; Jeffrey Edward Jenkins; Todd Richard Reily; Harold Ronald Villanueva Tagunicar; Michael Anthony Sciscenti
- Web page: https://patentgrep.com/patent/US10466858B2

## CPC

- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- A45C 11/00: Receptacles for purposes not provided for in groups A45C1/00-A45C9/00 (specially adapted for toiletry or cosmetic equipment A45D29/20, A45D44/18; travelling sewing kits A45F3/48)
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 27/017: {Head mounted}
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/04812: Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
- G06F 3/04817: using icons (graphical or visual programming using iconic symbols G06F8/34)
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/0485: Scrolling or panning
- G06F 3/0487: using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 21/71: to assure secure computing or processing of information
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04L 12/1822: {Conducting the conference, e.g. admission, detection, selection or grouping of participants, correlating users to one or more conference sessions, prioritising transmission}
- H04L 67/52: specially adapted for the location of the user terminal
- H04M 1/72454: according to context-related or environment-related conditions
- H04N 23/51: Housings
- H04N 23/61: based on recognised objects
- H04N 23/631: {Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters}
- H04N 23/635: {Region indicators; Field of view indicators}
- H04W 4/08: User group management
- H04W 12/084: using delegated authorisation, e.g. open authorisation [OAuth] protocol
- H04W 76/10: Connection setup
- A45C 11/002: {for storing portable handheld communication devices, e.g. pagers or smart phones}
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0141: {characterised by the informative content of the display}
- G02B 2027/0178: {Eyeglass type (eyeglass details G02C)}
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06F 3/0486: Drag-and-drop
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- A45C: PURSES; LUGGAGE; HAND CARRIED BAGS
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

Systems and methods allow a user to interact with an augmented reality device. In one implementation, a wearable device for providing a virtual menu to a user includes a display; at least one sensor configured to provide an output indicative of a viewing direction the user; and at least one processing device. The at least one processing device is configured to cause a virtual menu to be shown on the display; monitor a viewing direction of the user based on the output of the at least one sensor; determine, based on the monitored viewing direction, whether the user is looking in a direction of a selectable element of the virtual menu; determine an amount of time that the user looks in the direction of the selectable element; and cause at least one action associated with the selectable element if the amount of time exceeds a predetermined dwell time threshold.

## Claims

### Claim 1

A wearable device for providing a virtual menu to a user, the wearable device comprising:
a display;
at least one sensor configured to provide an output indicative of an orientation of a head of the user; and
at least one processing device configured to:
monitor the orientation of the head of the user based on the output of the at least one sensor;
track a pattern of a movement of the head of the user;
cause the virtual menu to be shown on the display after detecting, via the at least one sensor, that the pattern of the movement of the head of the user matches a predetermined pattern, the virtual menu including at least one selectable element;
cause an indicator of a predetermined dwell time threshold representing a predetermined time duration to be shown on the display, the predetermined time duration being associated with selecting the at least one selectable element on the display;
determine, based on the orientation of the head of the user, whether the user is looking in a direction of the at least one selectable element of the virtual menu;
determine, based on the output indicative of the orientation of the head of the user, an amount of time that the user looks in the direction of the at least one selectable element of the virtual menu;
cause at least one action associated with the at least one selectable element of the virtual menu if the amount of time exceeds the predetermined dwell time threshold;
receive a user selection of one of a menu locked mode and a menu unlocked mode;
in response to the user's selection of the menu locked mode, fix the virtual menu relative to the display such that the virtual menu does not move relative to the display as the head of the user moves;
in response to the user's selection of the menu unlocked mode, fix the virtual menu relative to a real-world coordinate system such that the virtual menu moves relative to the display as the head of the user moves;
in response to receiving a user's request to unlock the wearable device, cause the display to display an image including an arrangement of points for entering a password and a cursor movable based on the output of the at least one sensor;
in response to detecting that the cursor is moved to a first point among the arrangement of points, record a first dwell time associated with the first point;
in response to detecting the cursor is moved from the first point to a second point among the arrangement of points, record a second dwell time associated with the second point;
determine whether a sequence of the first point and the second point and their respectively associated first dwell time and second dwell time matches a password sequence pattern, the password sequence pattern including a sequence of points and their respectively associated dwell times; and
in response to determining that the sequence of the first point and the second point and their respectively associated first dwell time and second dwell time matches the password sequence pattern, unlock the wearable device.

### Claim 2 (depends on 1)

The wearable device of claim 1, wherein the display includes a lens in glasses.

### Claim 3 (depends on 1)

The wearable device of claim 1, wherein the at least one selectable element comprises one of an icon, a check-box, and text.

### Claim 4 (depends on 1)

The wearable device of claim 1, wherein the virtual menu is semi-transparent.

### Claim 5 (depends on 4)

The wearable device of claim 4, wherein the transparency of the virtual menu is adjustable based on an input received by the at least one processing device from the user.

### Claim 6 (depends on 1)

The wearable device of claim 1, wherein the predetermined dwell time threshold is within a range of about 0.1 seconds to about 5 seconds.

### Claim 7 (depends on 1)

The wearable device of claim 1, wherein the predetermined dwell time threshold is within a range of about 0.25 seconds to about 2 seconds.

### Claim 8 (depends on 1)

The wearable device of claim 1, wherein the predetermined dwell time threshold is about 0.5 seconds.

### Claim 9 (depends on 1)

The wearable device of claim 1, wherein the predetermined dwell time threshold is selectable by the user.

### Claim 10 (depends on 1)

The wearable device of claim 1, wherein the at least one action associated with the at least one selectable element comprises expanding the at least one selectable element.

### Claim 11 (depends on 1)

The wearable device of claim 1, wherein the at least one action associated with the at least one selectable element comprises opening an application associated with the at least one selectable element.

### Claim 12 (depends on 1)

The wearable device of claim 1, wherein the at least one selectable element comprises text and the at least one action associated with the at least one selectable element comprises providing an audible output of the text.

### Claim 13 (depends on 12)

The wearable device of claim 12, further comprising a speaker, wherein the audible output of the text is provided to the user via the speaker.

### Claim 14 (depends on 13)

The wearable device of claim 13, wherein the speaker is included in an earbud configured for placement inside the user's ear.

### Claim 15

A method for providing a virtual menu to a user of a wearable device, the method comprising:
monitoring an orientation of a head of the user based on an output of at least one sensor configured to track the orientation of the head of the user;
tracking a pattern of a movement of the head of the user;
causing the virtual menu to be shown on a display of the wearable device after detecting, via the at least one sensor, that the pattern of the movement of the head of the user matches a predetermined pattern, wherein the virtual menu includes at least one selectable element;
causing an indicator of a predetermined dwell time threshold representing a predetermined time duration to be shown on the display, the predetermined time duration being associated with selecting the at least one selectable element on the display;
determining, based on the orientation of the head of the user, whether the user is looking in a direction of the at least one selectable element of the virtual menu;
determining, based on the output indicative of the orientation of the head of the user, an amount of time that the user looks in the direction of the at least one selectable element of the virtual menu;
causing at least one action associated with the at least one selectable element of the virtual menu if the amount of time exceeds the predetermined dwell time threshold;
receiving a user selection of one of a menu locked mode and a menu unlocked mode;
in response to the user's selection of the menu locked mode, fixing the virtual menu relative to the display such that the virtual menu does not move relative to the display as the head of the user moves;
in response to the user's selection of the menu unlocked mode, fixing the virtual menu relative to a real-world coordinate system such that the virtual menu moves relative to the display as the head of the user moves;
in response to receiving a user's request to unlock the wearable device, causing the display to display an image including an arrangement of points for entering a password and a cursor movable based on the output of the at least one sensor;
in response to detecting that the cursor is moved to a first point among the arrangement of points, recording a first dwell time associated with the first point;
in response to detecting the cursor is moved from the first point to a second point among the arrangement of points, recording a second dwell time associated with the second point;
determining whether a sequence of the first point and the second point and their respectively associated first dwell time and second dwell time matches a password sequence pattern, the password sequence pattern including a sequence of points and their respectively associated dwell times; and
in response to determining that the sequence of the first point and the second point and their respectively associated first dwell time and second dwell time matches the password sequence pattern, unlocking the wearable device.

### Claim 16 (depends on 15)

The method of claim 15, wherein the wearable device includes the at least one sensor.

### Claim 17 (depends on 15)

The method of claim 15, wherein the at least one selectable element comprises one of an icon, a check-box, and text.

### Claim 18 (depends on 15)

The method of claim 15, wherein the virtual menu is semi-transparent.

### Claim 19 (depends on 18)

The method of claim 18, wherein the transparency of the virtual menu is adjustable based on an input received by the at least one processing device from the user.

### Claim 20 (depends on 15)

The method of claim 15, wherein the predetermined dwell time threshold is selected by the user.

### Claim 21 (depends on 15)

The method of claim 15, wherein the at least one action associated with the at least one selectable element comprises expanding the at least one selectable element.

### Claim 22 (depends on 15)

The method of claim 15, wherein the at least one action associated with the at least one selectable element comprises opening an application associated with the at least one selectable element.

### Claim 23 (depends on 15)

The method of claim 15, wherein the at least one selectable element comprises text and the at least action associated with the at least one selectable element comprises providing an audible output of the text.

### Claim 24

A non-transitory computer-readable storage medium storing program instructions executable by at least one processor to perform a method for providing a virtual menu to a user of a wearable device, the method comprising:
monitoring an orientation of a head of the user based on an output of at least one sensor configured to track the orientation of the head of the user;
tracking a pattern of a movement of the head of the user;
causing the virtual menu to be shown on a display of the wearable device after detecting, via the at least one sensor, that the pattern of the movement of the head of the user matches a predetermined pattern, wherein the virtual menu includes at least one selectable element;
causing an indicator of a predetermined dwell time threshold representing a predetermined time duration to be shown on the display, the predetermined time duration being associated with selecting the at least one selectable element on the display;
determining, based on the orientation of the head of the user, whether the user is looking in a direction of the at least one selectable element of the virtual menu;
determining, based on the output indicative of the orientation of the head of the user, an amount of time that the user looks in the direction of the at least one selectable element of the virtual menu;
causing at least one action associated with the at least one selectable element of the virtual menu if the amount of time exceeds the predetermined dwell time threshold;
receiving a user selection of one of a menu locked mode and a menu unlocked mode;
in response to the user's selection of the menu locked mode, fixing the virtual menu relative to the display such that the virtual menu does not move relative to the display as the head of the user moves;
in response to the user's selection of the menu unlocked mode, fixing the virtual menu relative to a real-world coordinate system such that the virtual menu moves relative to the display as the head of the user moves;
in response to receiving a user's request to unlock the wearable device, causing the display to display an image including an arrangement of points for entering a password and a cursor movable based on the output of the at least one sensor;
in response to detecting that the cursor is moved to a first point among the arrangement of points, recording a first dwell time associated with the first point;
in response to detecting the cursor is moved from the first point to a second point among the arrangement of points, recording a second dwell time associated with the second point;
determining whether a sequence of the first point and the second point and their respectively associated first dwell time and second dwell time matches a password sequence pattern, the password sequence pattern including a sequence of points and their respectively associated dwell times; and
in response to determining that the sequence of the first point and the second point and their respectively associated first dwell time and second dwell time matches the password sequence pattern, unlocking the wearable device.

### Claim 25 (depends on 1)

The wearable device of claim 1, wherein the at least one processing device is further configured to:
cause the display to display a first number at the first point, the first number representing the first dwell time; and
cause the display to display a second number at the second point, the second number representing the second dwell time.

### Claim 26 (depends on 1)

The wearable device of claim 1, wherein the at least one processing device is further configured to:
cause the display the display the first point in a first color which represents the first dwell time; and
cause the display the display the second point in a second color which represents the second dwell time.