Surface-based user input for extended reality

An extended reality (XR) device comprising:

• at least one processor;
• and
• memory storing instructions that, when executed by the at least one processor, cause the XR device to perform operations comprising:
• tracking a hand of a user across multiple image frames to obtain positions of one or more parts of the hand in a real-world environment;
• detecting an input plane associated with a physical surface in the real-world environment;
• projecting the positions onto the input plane;
• monitoring an input state with respect to the input plane to:
• identify when the user is providing user input via the input plane, and
• differentiate between ongoing user input and finalized user input;
• recording, from among the projected positions, multiple projected positions as input data based on the monitoring of the input state, the recording of the multiple projected positions comprising:
• while the input state indicates that the user input is ongoing, recording the multiple projected positions as the user input for a single user interaction,
• detecting a transition in the input state indicating that the user input is finalized, and
• based on detecting the transition, consolidating the recorded multiple projected positions for the single user interaction for unified interpretation;
• processing the input data to interpret the user input;
• and
• executing an action based on the interpreted user input.

One or more non-transitory computer-readable storage media, the one or more non-transitory computer-readable storage media including instructions that when executed by at least one processor of an extended reality (XR) device cause the XR device to perform operations comprising:

• tracking a hand of a user across multiple image frames to obtain positions of one or more parts of the hand in a real-world environment;
• detecting an input plane associated with a physical surface in the real-world environment;
• projecting the positions onto the input plane;
• monitoring an input state with respect to the input plane to:
• identify when the user is providing user input via the input plane, and
• differentiate between ongoing user input and finalized user input;
• recording, from among the projected positions, multiple projected positions as input data based on the monitoring of the input state, the recording of the multiple projected positions comprising:
• while the input state indicates that the user input is ongoing, recording the multiple projected positions as the user input for a single user interaction,
• detecting a transition in the input state indicating that the user input is finalized, and
• based on detecting the transition, consolidating the recorded multiple projected positions for the single user interaction for unified interpretation;
• processing the input data to interpret the user input;
• and
• executing an action based on the interpreted user input.