# US 2025/0094116 A1 · Determining input modality of head-mounted device based on input modality of handheld device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094116 A1
- Document ID: US20250094116A1
- Status: Published application
- Kind: A1
- Application number: 18/884,920
- Filed: 2024-09-13
- Published: 2025-03-20
- Earliest claimed: 2023-09-14
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Ruofei Du; David Kim; Fengyuan Zhu; Daniel Kalmar
- Web page: https://patentgrep.com/patent/US20250094116A1

## CPC

- G06F 3/1454: {involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay}
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06F 3/04886: by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
- G06F 2203/04802: 3D-info-object: information is displayed on the internal or external surface of a three dimensional manipulable object, e.g. on the faces of a cube that can be rotated by the user
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method can include determining, by a head-mounted device mounted on a head of a user, a number of degrees of freedom of an input modality of an auxiliary device; determining an input modality of the head-mounted device based on the number of degrees of freedom of the input modality of the auxiliary device; and presenting the input modality of the head-mounted device to the user.

## Claims

### Claim 1

A method comprising:
determining, by a head-mounted device mounted on a head of a user, a number of degrees of freedom of an input modality of an auxiliary device;
determining an input modality of the head-mounted device based on the number of degrees of freedom of the input modality of the auxiliary device; and
presenting the input modality of the head-mounted device to the user.

### Claim 2 (depends on 1)

The method of claim 1, wherein determining the number of degrees of freedom of the input modality includes:
generating computer-executable code based on at least one image of the input modality of the auxiliary device, the computer-executable code being configured to generate the input modality of the auxiliary device; and
determining the number of degrees of freedom of the input modality of the auxiliary device based on the computer-executable code.

### Claim 3 (depends on 1)

The method of claim 1, wherein the auxiliary device includes a handheld device.

### Claim 4 (depends on 1)

The method of claim 1, wherein determining the number of degrees of freedom of the input modality includes:
generating computer-executable code based on at least one image of the input modality of the auxiliary device, the computer-executable code being configured to generate the input modality of the auxiliary device;
determining, by a language model, a classification of the input modality of the auxiliary device based on the computer-executable code; and
determining the number of degrees of freedom of the input modality of the auxiliary device based on the classification.

### Claim 5 (depends on 1)

The method of claim 1, wherein the input modality of the head-mounted device has a same number of degrees of freedom as the input modality of the auxiliary device.

### Claim 6 (depends on 1)

The method of claim 1, wherein:
the input modality of the auxiliary device includes a selection of a button presented by a display included in the auxiliary device; and
the input modality of the head-mounted device includes a selection of an icon presented by the head-mounted device.

### Claim 7 (depends on 1)

The method of claim 1, wherein:
the input modality of the auxiliary device includes a swipe on a display included in the auxiliary device; and
the input modality of the head-mounted device includes the head-mounted device capturing a gesture of a hand of the user.

### Claim 8 (depends on 1)

The method of claim 1, wherein:
the input modality of the auxiliary device includes typing onto a soft keyboard presented by a touchscreen display included in the auxiliary device; and
the input modality of the head-mounted device includes the head-mounted device capturing typing of a hand of the user onto a virtual keyboard presented by the head-mounted device.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
processing input via the input modality of the head-mounted device to generate an instruction; and
sending the instruction to the auxiliary device.

### Claim 10

A non-transitory computer-readable storage medium comprising instructions stored thereon that, when executed by at least one processor, are configured to cause a computing device to:
determine, by a head-mounted device mounted on a head of a user, a number of degrees of freedom of an input modality of an auxiliary device held by a hand of the user;
determine an input modality of the head-mounted device based on the number of degrees of freedom of the input modality of the auxiliary device; and
present the input modality of the head-mounted device to the user.

### Claim 11 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, wherein determining the number of degrees of freedom of the input modality includes:
generating computer-executable code based on at least one image of the input modality of the auxiliary device, the computer-executable code being configured to generate the input modality of the auxiliary device; and
determining the number of degrees of freedom of the input modality of the auxiliary device based on the computer-executable code.

### Claim 12 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the auxiliary device includes a handheld device.

### Claim 13 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, wherein determining the number of degrees of freedom of the input modality includes:
generating computer-executable code based on at least one image of the input modality of the auxiliary device, the computer-executable code being configured to generate the input modality of the auxiliary device;
determining, by a language model, a classification of the input modality of the auxiliary device based on the computer-executable code; and
determining the number of degrees of freedom of the input modality of the auxiliary device based on the classification.

### Claim 14 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, wherein the input modality of the head-mounted device has a same number of degrees of freedom as the input modality of the auxiliary device.

### Claim 15 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, wherein:
the input modality of the auxiliary device includes a selection of a button presented by a display included in the auxiliary device; and
the input modality of the head-mounted device includes a selection of an icon presented by the head-mounted device.

### Claim 16 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, wherein:
the input modality of the auxiliary device includes a swipe on a display included in the auxiliary device; and
the input modality of the head-mounted device includes the head-mounted device capturing a gesture of a hand of the user.

### Claim 17 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, wherein:
the input modality of the auxiliary device includes typing onto a soft keyboard presented by a touchscreen display included in the auxiliary device; and
the input modality of the head-mounted device includes the head-mounted device capturing typing of a hand of the user onto a virtual keyboard presented by the head-mounted device.

### Claim 18 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, wherein the instructions are further configured to cause the computing device to:
process input via the input modality of the head-mounted device to generate an instruction; and
send the instruction to the auxiliary device.

### Claim 19

A method comprising:
generating, by a head-mounted device within an extended Reality (XR) environment, a virtual display, the virtual display being based on a touchscreen display presented by an auxiliary device held by a user who is wearing the head-mounted device;
generating, within the XR environment, a virtual hand based on first image data captured by a camera, the camera being included in the head-mounted device;
presenting, within the XR environment, the virtual hand contacting the virtual display based on second image data captured by the camera;
determining at least one of a time or location of contact by a hand of the user onto the touchscreen display based on a signal received from the auxiliary device;
determining an adjustment based on the at least one of the time or location of the contact and the presentation of the virtual hand contacting the virtual display; and
presenting, within the XR environment, the virtual hand contacting the virtual display based on third image data captured by the camera and the adjustment.

### Claim 20 (depends on 19)

The method of claim 19, wherein the virtual display streams content presented by the touchscreen display.
