# US 2025/0094041 A1 · Smart Character Suggestion via XR Cubic Keyboard on Head-mounted Devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094041 A1
- Document ID: US20250094041A1
- Status: Published application
- Kind: A1
- Application number: 18/469,368
- Filed: 2023-09-18
- Published: 2025-03-20
- Earliest claimed: 2023-09-18
- Applicants: Meta Platforms, Inc.
- Inventors: Jiayang Xu
- Web page: https://patentgrep.com/patent/US20250094041A1

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/04886: by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection}
- G06F 3/0233: {Character input methods}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

In one embodiment, a method includes receiving a first user input from a user from a client system comprising a head-mounted extended-reality (XR) device, determining the user's intent to activate an XR cubic keyboard based on the first user input, rendering the XR cubic keyboard via XR displays of the head-mounted XR device, wherein the XR cubic keyboard comprises input areas representing respective characters in a three-dimensional (3D) space, and wherein the input areas are reachable by respective vectors from a centroid of the XR cubic keyboard in the 3D space, receiving a second user input comprising a hand movement of the user along a direction of a first vector from the centroid of the XR cubic keyboard in the 3D space, determining a first character that the user intended to input, and rendering an indication of the first character via the XR displays.

## Claims

### Claim 1

A method comprising, by one or more computing systems:
receiving, from a client system comprising a head-mounted extended-reality (XR) device, a first user input from a user;
determining, based on the first user input, an intent of the user to activate an XR cubic keyboard;
rendering, via one or more XR displays of the head-mounted XR device, the XR cubic keyboard, wherein the XR cubic keyboard comprises a plurality of input areas representing a plurality of characters, respectively, in a three-dimensional (3D) space, and wherein the plurality of input areas are reachable by a plurality of vectors, respectively, from a centroid of the XR cubic keyboard in the 3D space;
receiving, a second user input from the user, wherein the second user input comprises a hand movement of the user along a direction of a first vector of the plurality of vectors from the centroid of the XR cubic keyboard in the 3D space;
determining, based only on the direction of the first vector with respect to the XR cubic keyboard in the 3D space, a first character of the plurality of characters that the user intended to input; and
rendering, via the one or more XR displays, an indication of the first character.

### Claim 2 (depends on 1)

The method of claim 1, wherein each of the plurality of characters comprises one or more of a letter, a number, a symbol, a word, a phrase, or an emoji.

### Claim 3 (depends on 1)

The method of claim 1, wherein the plurality of input areas comprise 26 cubis, and wherein the plurality of vectors comprise 26 vectors from the centroid of the XR cubic keyboard in the 3D space.

### Claim 4 (depends on 3)

The method of claim 3, wherein 9 first cubis of the 26 cubis are located on a top plane in the 3D space, wherein 8 second cubis of the 26 cubis are located on a middle plane in the 3D space, and wherein 9 third cubis of the 26 cubis are located on a bottom plane in the 3D space.

### Claim 5 (depends on 3)

The method of claim 3, wherein the hand movement of the user is along a direction of a first vector of the 26 vectors from the centroid of the XR cubic keyboard in the 3D space.

### Claim 6 (depends on 1)

The method of claim 1, wherein the first user input comprises one or more of a gesture, a press on a button on the head-mounted XR device, or a voice input.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
determining the hand movement based on signals from an electromyography (EMG) wristband.

### Claim 8 (depends on 1)

The method of claim 1, wherein the head-mounted XR device is associated with one or more cameras, wherein the method further comprises:
receiving, from the one or more cameras, visual signals captured by the one or more cameras; and
determining, based on the visual signals by one or more machine-learning models, the hand movement.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
receiving, from the client system, a third user input via an electromyography (EMG) wristband, wherein the third user input comprises a gesture; and
determining, based on the gesture, the user confirms a selection of the first character.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
rendering a confirmation with the user of the selected first character, wherein the rendering is based on one or more of a visual display of the selected first character on the one or more XR displays, a haptic feedback, or an audio feedback.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
generating, based on the first character, one or more candidate commands for the user; and
rendering one or more of the candidate commands, wherein the rendering is based on one or more of a visual display of the one or more of the candidate commands on the one or more XR displays or a readout of the one or more of the candidate commands.

### Claim 12 (depends on 11)

The method of claim 11, further comprising:
receiving, from the client system, a user selection of a first candidate command of the rendered candidate commands;
determining a first task corresponding to the first candidate command; and
executing the first task.

### Claim 13 (depends on 1)

The method of claim 1, wherein the client system further comprises an electromyography (EMG) wristband, and wherein the EMG wristband comprises one or more inertial measurement unit (IMU) sensors, wherein the method further comprises:
determining the direction of the first vector with respect to the XR cubic keyboard in the 3D space based on signals from the one or more IMU sensors.

### Claim 14 (depends on 1)

The method of claim 1, wherein the client system further comprises an electromyography (EMG) wristband, and wherein the first and second user inputs are received via the EMG wristband.

### Claim 15

One or more computer-readable non-transitory non-volatile storage media embodying software that is operable when executed to:
receive, from a client system comprising a head-mounted extended-reality (XR) device, a first user input from a user;
determine, based on the first user input, an intent of the user to activate an XR cubic keyboard;
render, via one or more XR displays of the head-mounted XR device, the XR cubic keyboard, wherein the XR cubic keyboard comprises a plurality of input areas representing a plurality of characters, respectively, in a three-dimensional (3D) space, and wherein the plurality of input areas are reachable by a plurality of vectors, respectively, from a centroid of the XR cubic keyboard in the 3D space;
receive, a second user input from the user, wherein the second user input comprises a hand movement of the user along a direction of a first vector of the plurality of vectors from the centroid of the XR cubic keyboard in the 3D space;
determine, based only on the direction of the first vector with respect to the XR cubic keyboard in the 3D space, a first character of the plurality of characters that the user intended to input; and
render, via the one or more XR displays, an indication of the first character.

### Claim 16 (depends on 15)

The media of claim 15, wherein each of the plurality of characters comprises one or more of a letter, a number, a symbol, a word, a phrase, or an emoji.

### Claim 17 (depends on 15)

The media of claim 15, wherein the client system further comprises an electromyography (EMG) wristband, and wherein the EMG wristband comprises one or more inertial measurement unit (IMU) sensors, wherein the software is further operable when executed to:
determine the direction of the first vector with respect to the XR cubic keyboard in the 3D space based on signals from the one or more IMU sensors.

### Claim 18

A system comprising: one or more processors; and a non-transitory non-volatile memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
receive, from a client system comprising a head-mounted extended-reality (XR) device, a first user input from a user;
determine, based on the first user input, an intent of the user to activate an XR cubic keyboard;
render, via one or more XR displays of the head-mounted XR device, the XR cubic keyboard, wherein the XR cubic keyboard comprises a plurality of input areas representing a plurality of characters, respectively, in a three-dimensional (3D) space, and wherein the plurality of input areas are reachable by a plurality of vectors, respectively, from a centroid of the XR cubic keyboard in the 3D space;
receive, a second user input from the user, wherein the second user input comprises a hand movement of the user along a direction of a first vector of the plurality of vectors from the centroid of the XR cubic keyboard in the 3D space;
determine, based only on the direction of the first vector with respect to the XR cubic keyboard in the 3D space, a first character of the plurality of characters that the user intended to input; and
render, via the one or more XR displays, an indication of the first character.

### Claim 19 (depends on 18)

The system of claim 18, wherein each of the plurality of characters comprises one or more of a letter, a number, a symbol, a word, a phrase, or an emoji.

### Claim 20 (depends on 18)

The system of claim 18, wherein the client system further comprises an electromyography (EMG) wristband, and wherein the EMG wristband comprises one or more inertial measurement unit (IMU) sensors, wherein the processors are further operable when executing the instructions to:
determine the direction of the first vector with respect to the XR cubic keyboard in the 3D space based on signals from the one or more IMU sensors.