# US 2021/0081052 A1 · User interface control based on elbow-anchored arm gestures

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0081052 A1
- Document ID: US20210081052A1
- Status: Published application
- Kind: A1
- Application number: 17/024,633
- Filed: 2020-09-17
- Published: 2021-03-18
- Earliest claimed: 2019-09-17
- Assignees: Huawei Technologies Co., Ltd.
- Inventors: Gaganpreet SINGH; Rafael VERAS GUIMARAES; Wei ZHOU; Pourang Polad IRANI; Farzin FARHADI-NIAKI; Wei LI
- Web page: https://patentgrep.com/patent/US20210081052A1

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/014: {Hand-worn input/output arrangements, e.g. data gloves}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

User interface control based on elbow-anchored arm gestures is disclosed. In one aspect, there is provided a method of user interface control on a computing device based on elbow-anchored arm gestures. A visual user interface (VUI) screen is displayed on a display of a computing device. The VUI screen comprises a plurality of VUI elements arranged in a plurality of VUI element levels. Each VUI element level comprising one or more VUI elements. A spatial location of an elbow of an arm of a user and a spatial location of a wrist of the arm of the user are determined. A three-dimensional (3D) arm vector extending from the spatial location of the elbow to the spatial location of the wrist is then determined. A VUI element in the VUI screen corresponding to the 3D arm vector is then determined based on a predetermined 3D spatial mapping between 3D arm vectors and VUI elements for the VUI screen.

## Claims

### Claim 1

A method of user interface control on a computing device based on elbow-anchored arm gestures, comprising:
causing a visual user interface (VUI) screen to be displayed on a display of the computing device, wherein the VUI screen comprises a plurality of VUI elements arranged in a plurality of VUI element levels, each VUI element level comprising one or more VUI elements;
determining, based on sensor data, a spatial location of an elbow of an arm of a user;
determining, based on sensor data, a spatial location of a wrist of the arm of the user;
determining a three-dimensional (3D) arm vector extending from the spatial location of the elbow to the spatial location of the wrist; and
determining a VUI element in the VUI screen corresponding to the 3D arm vector based on a predetermined 3D spatial mapping between 3D arm vectors and VUI elements for the VUI screen.

### Claim 2 (depends on 1)

The method of claim 1, wherein determining the VUI element in the VUI screen corresponding to the 3D arm vector comprises:
determining a forward inclination angle formed between the 3D arm vector and a horizontal reference plane;
determining a lateral inclination angle formed between the 3D arm vector and a vertical reference plane; and
determining a VUI element in the VUI screen based on the forward inclination angle and the lateral inclination angle.

### Claim 3 (depends on 2)

The method of claim 2, wherein determining the VUI element in the VUI screen comprises:
determining a corresponding VUI element level of the VUI screen among the plurality of VUI element levels of the VUI screen based on the forward inclination angle; and
determining a corresponding VUI element in the determined VUI element level of the VUI screen based on the lateral inclination angle.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
prior to causing the VUI screen to be displayed:
in response to input to display the VUI screen:
determining a spatial location of an elbow of an arm of a user;
determining a spatial location of a wrist of the arm of the user;
determining a 3D arm vector extending from the spatial location of the elbow to the spatial location of the wrist;
determining an offset angle between the 3D arm vector and a vertical centerline from the spatial location of the elbow; and
generating the VUI screen offset from the centerline of the display of the computing device in response to a determination that the offset angle is greater than or equal to a threshold offset angle.

### Claim 5 (depends on 1)

The method of claim 1, wherein upper VUI element levels in VUI screen relative to a screen orientation of the VUI screen have fewer VUI elements than lower VUI element levels in VUI screen relative to a screen orientation of the VUI screen.

### Claim 6 (depends on 1)

The method of claim 1, wherein the plurality of VUI elements are arranged in spherical grid.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
selecting the determined VUI element; and
visually emphasizing the selected VUI element.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
performing an executable action corresponding to the selected VUI element in response to the detection of confirmation input.

### Claim 9 (depends on 8)

The method of claim 8, wherein the confirmation input is a designated hand gesture.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
performing an executable action corresponding to the determined VUI element.

### Claim 11

A computing device, comprising:
a display;
a processor coupled the display, wherein the processor is configured to:
cause a visual user interface (VUI) screen to be displayed on the display, wherein the VUI screen comprises a plurality of VUI elements arranged in a plurality of VUI element levels, each VUI element level comprising one or more VUI elements;
determine, based on sensor data, a spatial location of an elbow of an arm of a user based on sensor data;
determine, based on sensor data, a spatial location of a wrist of the arm of the user;
determine a three-dimensional (3D) arm vector extending from the spatial location of the elbow to the spatial location of the wrist; and
determine a VUI element in the VUI screen corresponding to the 3D arm vector based on a predetermined 3D spatial mapping between 3D arm vectors and VUI elements for the VUI screen.

### Claim 12 (depends on 11)

The computing device of claim 11, wherein the processor is configured to determine the VUI element in the VUI screen corresponding to the 3D arm vector by:
determining a forward inclination angle formed between the 3D arm vector and a horizontal reference plane;
determining a lateral inclination angle formed between the 3D arm vector and a vertical reference plane; and
determining a VUI element in the VUI screen based on the forward inclination angle and the lateral inclination angle.

### Claim 13 (depends on 12)

The computing device of claim 12, wherein the processor is configured to determine the VUI element in the VUI screen by:
determining a corresponding VUI element level of the VUI screen among the plurality of VUI element levels of the VUI screen based on the forward inclination angle; and
determining a corresponding VUI element in the determined VUI element level of the VUI screen based on the lateral inclination angle.

### Claim 14 (depends on 11)

The computing device of claim 11, wherein the processor is further configured to:
prior to causing the VUI screen to be displayed:
in response to input to display the VUI screen:
determine a spatial location of an elbow of an arm of a user;
determine a spatial location of a wrist of the arm of the user;
determine a 3D arm vector extending from the spatial location of the elbow to the spatial location of the wrist;
determine an offset angle between the 3D arm vector and a vertical centerline from the spatial location of the elbow; and
generate the VUI screen offset from the centerline of the display of the computing device in response to a determination that the offset angle is greater than or equal to a threshold offset angle.

### Claim 15 (depends on 11)

The computing device of claim 11, wherein upper VUI element levels in VUI screen relative to a screen orientation of the VUI screen have fewer VUI elements than lower VUI element levels in VUI screen relative to a screen orientation of the VUI screen.

### Claim 16 (depends on 11)

The computing device of claim 11, wherein the plurality of VUI elements are arranged in spherical grid.

### Claim 17 (depends on 11)

The computing device of claim 11, wherein the processor is further configured to:
select the determined VUI element; and
visually emphasize the selected VUI element.

### Claim 18 (depends on 11)

The computing device of claim 11, wherein the processor is further configured to:
perform an executable action corresponding to the selected VUI element in response to the detection of confirmation input.

### Claim 19 (depends on 11)

The computing device of claim 11, wherein the processor is further configured to:
perform an executable action corresponding to the determined VUI element.

### Claim 20

A non-transitory machine-readable medium having tangibly stored thereon executable instructions that, in response to execution by a processor of a computing device, cause the processor to:
cause a visual user interface (VUI) screen to be displayed on a display of the computing device, wherein the VUI screen comprises a plurality of VUI elements arranged in a plurality of VUI element levels, each VUI element level comprising one or more VUI elements;
determine, based on sensor data, a spatial location of an elbow of an arm of a user;
determine, based on sensor data, a spatial location of a wrist of the arm of the user;
determine a three-dimensional (3D) arm vector extending from the spatial location of the elbow to the spatial location of the wrist; and
determine a VUI element in the VUI screen corresponding to the 3D arm vector based on a predetermined 3D spatial mapping between 3D arm vectors and VUI elements for the VUI screen.
