# US 2024/0004545 A1 · Intelligent keyboard attachment for mixed reality input

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0004545 A1
- Document ID: US20240004545A1
- Status: Published application
- Kind: A1
- Application number: 18/468,682
- Filed: 2023-09-15
- Published: 2024-01-04
- Earliest claimed: 2021-11-04
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Andrew Jackson KLEIN; Hendry EFFENDY; Ethan Harris ARNOWITZ; Jonathon Burnham COBB; Melissa HELLMUND VEGA; Stuart John MAYHEW; Jeremy Bruce KERSEY
- Web page: https://patentgrep.com/patent/US20240004545A1

## CPC

- G06F 3/04886: by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0236: {using selection techniques to select from displayed items}
- G06F 3/0416: {Control or interface arrangements specially adapted for digitisers}
- G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 3/0426: {tracking fingers with respect to a virtual keyboard projected or printed on the surface (virtual keyboards on touch screens G06F3/04886)}
- G06F 2203/04101: 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems and methods for attaching a virtual input device to a virtual object in a mixed reality (MR) environment are provided. The system includes a memory, a processor communicatively coupled to the memory, and a display device. The display device is configured to display a MR environment provided by at least one application implemented by the processor. The mixed reality environment includes a virtual object corresponding to an application, and a virtual input device. The at least one application docks the virtual input device to the virtual object with an offset relative to the virtual object.

## Claims

### Claim 1

A system comprising:
a memory storing applications;
a processor communicatively coupled to the memory; and
a display device configured to display a mixed reality environment provided by one or more of the applications implemented by the processor, wherein the mixed reality environment includes:
a first virtual object corresponding to a first application;
a virtual input device, wherein the first application docks the virtual input device to the first virtual object with an offset relative to the first virtual object; and
a second virtual object corresponding to a second application, wherein responsive to user input indicating an intent to interact with the second virtual object, the second application docks the virtual input device with an offset relative to the second virtual object.

### Claim 2 (depends on 1)

The system of claim 1, wherein the offset of the virtual input device to the first virtual object includes a predetermined distance between the virtual input device and the first virtual object and a predetermined angle between the virtual input device and the first virtual object.

### Claim 3 (depends on 1)

The system of claim 1, wherein the offset of the virtual input device to the first virtual object includes a dynamic distance between the virtual input device and the first virtual object and a dynamic angle between the virtual input device and the first virtual object.

### Claim 4 (depends on 1)

The system of claim 1, wherein the first application docks the virtual input device to the first virtual object by creating a hierarchical positioning transformational relationship between the first virtual object and the virtual input device.

### Claim 5 (depends on 1)

The system of claim 1, wherein the display device presents a field of view including at least a portion of the mixed reality environment, the field of view including the first virtual object and the virtual input device.

### Claim 6 (depends on 5)

The system of claim 5, wherein the processor is configured to:
detect an update to the field of view based at least on a movement of the display device, the update removing the virtual object and the virtual input device from the field of view, and
return the virtual object and the virtual input device to the updated field of view while maintaining the offset between the virtual object and the virtual input device.

### Claim 7 (depends on 1)

The system of claim 1, wherein the first virtual object and the virtual input device, including the offset, are maintained in an original position proximate to an original position of a user based at least on a movement of the display device indicating movement of the user to another position.

### Claim 8 (depends on 1)

The system of claim 1, wherein responsive to user input indicating an intent to transition back to interact with the first virtual object, re-positioning the virtual input device with the offset relative to the first virtual object.

### Claim 9 (depends on 1)

The system of claim 1, wherein the user input indicating the intent is one of: an input received via the virtual input device, a gaze pattern determined by the display device, or a selection of a command on the first virtual object.

### Claim 10

A computer-implemented method comprising:
displaying, by a display device, a mixed reality environment,
positioning a first virtual object within the mixed reality environment, the first virtual object corresponding to a first application;
docking a virtual input device to the first virtual object with an offset relative to the first virtual object;
receiving a user input indicating an intent to interact with a second virtual object corresponding to a second application; and
responsive to the user input, docking the virtual input device to the second virtual object with an offset relative to the second virtual object.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10, wherein the offset of the virtual input device to the first virtual object includes a predetermined distance between the virtual input device and the first virtual object and a predetermined angle between the virtual input device and the first virtual object.

### Claim 12 (depends on 10)

The computer-implemented method of claim 10, wherein docking the virtual input device to the first virtual object further comprises creating a hierarchical positioning transformational relationship between the first virtual object and the virtual keyboard.

### Claim 13 (depends on 10)

The computer-implemented method of claim 10, wherein displaying the mixed reality environment further comprises:
displaying, by the display device, a field of view including at least a portion of the mixed reality environment, the field of view including the first virtual object and the virtual input device;
detecting an update to the field of view; and
maintaining the first virtual object and the virtual input device in the updated field of view.

### Claim 14 (depends on 10)

The computer-implemented method of claim 10, wherein positioning the first virtual object within the mixed reality environment further comprises:
positioning the first virtual object relative to an original position of a user associated with the display device;
detecting movement of the display device indicating movement of the user to another position; and
maintaining the positioning of the first virtual object relative to the original position of the user.

### Claim 15 (depends on 10)

The computer-implemented method of claim 10, further comprising:
receiving a user input indicating an intent to transition back to interact with the first virtual object; and
responsive to the user input, re-positioning the virtual input device to the first virtual object with the offset relative to the first virtual object.

### Claim 16

A head mounted device (HMD) comprising:
one or more lenses configured to display a mixed reality environment including a field of view, the field of view comprising at least a portion of the mixed reality environment that includes a first virtual object and a virtual input device, the virtual object representing a first application; and
at least one processor configured to:
determine a docking distance between the first virtual object and the virtual input device within the mixed reality environment,
determine a docking angle between the first virtual object and the virtual input device within the mixed reality environment,
dock the virtual input device to the first virtual object using the determined docking distance and the determined docking angle within the field of view;
receive a user input indicating an intent to interact with a second virtual object corresponding to a second application;
responsive to the user input, determine a second docking distance and a second docking angle between the second virtual object and the virtual input device; and
dock the virtual input device to the second virtual object with the determined second docking distance and the determined second docking angle within the field of view.

### Claim 17 (depends on 16)

The HMD of claim 16, wherein the at least one processor is further configured to:
detect a movement of the second virtual object within the field of view, and
control the virtual input device to move with the second virtual object and maintain the determined second docking distance and the determined second docking angle.

### Claim 18 (depends on 16)

The HMD of claim 16, wherein determining the docking distance and the docking angle further comprises the at least one processor further configured to:
position the first virtual object relative to an original position of a user associated with the HMD;
detect movement of the HMD indicating movement of the user to another position; and
maintain the position of the first virtual object relative to the original position of the user.

### Claim 19 (depends on 16)

The HMD of claim 16, wherein the at least one processor is further configured to:
detect a change in the field of view that causes the virtual input device and the second virtual object to be outside of the changed field of view, and
gradually reintroduce the second virtual object and the virtual input device into the changed field of view while maintaining the determined second docking distance and the determined second docking angle between the second virtual object and the virtual input device.

### Claim 20 (depends on 16)

The HMD of claim 16, wherein, to dock the virtual input device to the first virtual object, the at least one processor is further configured to:
create a hierarchical positioning transformational relationship between the first virtual object and the virtual input device.
