# US 2018/0321737 A1 · Integrated mixed-input system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0321737 A1
- Document ID: US20180321737A1
- Status: Published application
- Kind: A1
- Application number: 15/640,263
- Filed: 2017-06-30
- Published: 2018-11-08
- Earliest claimed: 2017-05-05
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Michel PAHUD; Eyal OFEK
- Web page: https://patentgrep.com/patent/US20180321737A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/0426: {tracking fingers with respect to a virtual keyboard projected or printed on the surface (virtual keyboards on touch screens G06F3/04886)}
- G06F 3/048: Interaction techniques based on graphical user interfaces [GUI]
- G06T 3/20: Linear translation of whole images or parts thereof, e.g. panning
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04886: by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

In various embodiments, methods and systems for implementing an integrated mixed-input system are provided. The integrated mixed-input system includes paired mixed-input devices for interacting and controlling virtual space input interfaces using real inputs and virtual inputs, sensors, and passive and active haptic feedback associated with the paired mixed-input devices. Real device space tracker data and virtual device space tracker data are accessed via the paired mixed-input devices to determine real input and virtual input that are processed to determine virtual space input. The real device space tracker data and virtual device space tracker data also are used to generate different interaction contexts. In one embodiment, integrated mixed-input system supports interface deviation, where a physical mixed-input device interface is a different size from a size of the virtual space input interface. The virtual space input is communicated to control the virtual space input interface.

## Claims

### Claim 1

One or more computer storage media having computer-executable instructions embodied thereon that, when executed, by one or more processors, cause the one or more processors to perform a method for implementing integrating mixed-input systems, the method comprising:
determining real input based on real device space tracker data associated with a physical mixed-input device;
determining virtual input based on virtual device space tracker data associated with a virtual mixed-input device, wherein a physical mixed-input device interface of the physical mixed-input device comprises a first interface size which is smaller than a second interface size, and wherein a virtual space input interface has the second interface size which is larger, wherein the virtual space input interface is dynamically shifted, such that, a target input portion of the virtual space input interface overlaps with the physical mixed-input device interface to receive the real input, wherein the virtual space input interface is generated as part of a wholly immersive virtual experience;
determining a virtual space input for the virtual space input interface based on the real input and the virtual input in an interaction context; and
communicating the virtual space input to control the virtual space input interface.

### Claim 2 (depends on 1)

The media of claim 1, wherein the physical mixed-input device is a mobile device having a display that displays the physical mixed-input device interface and wherein the virtual mixed-input device is a head-mounted display that displays the virtual space input interface.

### Claim 3 (depends on 1)

The media of claim 1, wherein the physical mixed-input device and the virtual mixed-input device are paired for determining the virtual space input for the virtual space input interface displayed via the virtual mixed-input device.

### Claim 4 (depends on 1)

The media of claim 1, further comprising detecting proximity or motion data of external objects to the physical mixed-input device or orientation of the physical mixed-input device to supplement determining of the interaction context.

### Claim 5 (depends on 1)

The media of claim 1, wherein the real input on the physical mixed-input device via the target input portion of the virtual space input interface causes at least the physical mixed-input device to generated haptic feedback.

### Claim 6 (depends on 1)

The media of claim 1, wherein the real input and the virtual input captured via corresponding real device sensors and virtual device sensors are integrated and processed in combination to determine the virtual space input for the virtual space input interface.

### Claim 7 (depends on 1)

The media of claim 1, wherein different combination of real input and virtual input trigger corresponding interaction contexts selected from one of the following:
a first interaction context comprising rendering a virtual representation of the physical mixed-input device coupled to a virtual second hand corresponding to a physical second hand coupled to the physical mixed-input device to receive input on the physical mixed-input device;
a second interaction context comprising rendering the virtual representation of the physical mixed-input device coupled to the virtual second hand and a distant virtual first hand corresponding to a physical first hand to receive input on the physical mixed-input device;
a third interaction context comprising rendering a virtual representation of the physical mixed-input device interface on the virtual representation of the physical mixed-input device on the virtual second hand corresponding to the physical second hand to receive input on the physical mixed-input device; and
a fourth interaction context comprising rendering the virtual representation of the physical mixed-input device coupled to the virtual second hand and a floating virtual space input interface and a floating virtual first hand corresponding to the physical first hand to receive input on the physical mixed-input device.

### Claim 8

An integrated mixed-input system comprising:
a physical mixed-input device configured to:
determine real input based on real device space tracker data associated with the physical mixed-input device, wherein the physical mixed-input device is a mobile device having a display that displays the physical mixed-input device interface;
a virtual mixed-input device configured to:
determine virtual input based on virtual device space tracker data associated with the virtual mixed-input device, wherein the virtual mixed-input device is a head-mounted display that displays the virtual space input interface;
an integrated processing component to:
generate an interaction context based on the real device space tracker data and the virtual device space tracker data for a virtual space input interface;
determine the virtual space input based on the real input and the virtual input in the interaction context; and
communicate the virtual space input to control the virtual space input interface.

### Claim 9 (depends on 8)

The system of claim 8, wherein the physical mixed-input device is a mobile device having a display that displays the physical mixed-input device interface, the mobile device operates an application the receives and processes the real input, and wherein the virtual mixed-input device is a head-mounted display that displays the virtual space input interface, the head-mounted display support gaze selection for receiving and processing virtual input.

### Claim 10 (depends on 8)

The system of claim 8, further comprising a hover sensor configured to detect proximity or motion data of external objects to the physical mixed-input device to receive real input or determine the virtual space input to cause navigation through different sets of characters available via the physical mixed-input device or the virtual mixed-input device.

### Claim 11 (depends on 8)

The system of claim 8, further comprising an integrated deviation manager configured to executed interface deviation for the virtual space input interface, wherein the physical mixed-input device interface comprises a first interface size which is different from a second interface, wherein the virtual space input interface has the second size.

### Claim 12 (depends on 11)

The system of claim 11, wherein the first interface size is smaller than the second interface size, such that, interface deviation comprises dynamically shifting the virtual space input interface, such that, a target input portion of the virtual space input interface overlaps with the physical mixed-input input device interface; or
wherein the first interface size is larger than the second interface size, such that, interface deviation comprises receiving the real input based on the physical mixed-input device interface being dynamically shifted, such that, a target input portion of the virtual space input interface overlaps with the physical mixed-input input device interface.

### Claim 13 (depends on 8)

The system of claim 8, further comprising a haptic feedback component configured to generate haptic feedback on the physical mixed-input device and the virtual mixed-input device based on receiving real input on the physical mixed-input device.

### Claim 14 (depends on 8)

The system of claim 8, wherein the integrated processing component is further to identify different combinations of real input and virtual input to trigger corresponding interaction contexts selected from one of the following:
a first interaction context comprising rendering a virtual representation of the physical mixed-input device coupled to a virtual second hand corresponding to a physical second hand coupled to the physical mixed-input device to receive input on the physical mixed-input device;
a second interaction context comprising rendering the virtual representation of the physical mixed-input device coupled to the virtual second hand and a distant virtual first hand corresponding to a physical first hand to receive input on the physical mixed-input device;
a third interaction context comprising rendering a virtual representation of the physical mixed-input device interface on the virtual representation of the physical mixed-input device on the virtual second hand corresponding to the physical second hand to receive input on the physical mixed-input device; and
a fourth interaction context comprising rendering the virtual representation of the physical mixed-input device coupled to the virtual second hand and a floating virtual space input interface and a floating virtual first hand corresponding to the physical first hand to receive input on the physical mixed-input device.

### Claim 15

A computer-implemented method for implementing integrating mixed-input systems, the method comprising:
determining real input based on real device space tracker data associated with a physical mixed-input device;
determining virtual input based on virtual device space tracker data associated with a virtual mixed-input device;
generating an interaction context based on the real device space tracker data and the virtual device space tracker data for a virtual space input interface, wherein the interaction context comprises a virtual representation of the physical mixed-input device coupled to a virtual second hand corresponding to a physical second hand;
determining a virtual space input for the virtual space input interface based on the real input and the virtual input via the interaction context; and
communicating the virtual space input to control the virtual space input interface.

### Claim 16 (depends on 15)

The method of claim 15, wherein the physical mixed-input device is a mobile device having a display that displays a physical mixed-input device interface and wherein the virtual mixed-input device is a head-mounted display that displays the virtual space input interface, wherein the physical mixed-input device and the virtual mixed-input device are paired for determining the virtual space input for the virtual space input interface displayed via the virtual mixed-input device.

### Claim 17 (depends on 15)

The method of claim 15, further comprising executing an interface deviation operation for the virtual space input interface, wherein a physical mixed-input device interface comprises a first interface size which is different from a second interface size, wherein the virtual input space interface has the second size.

### Claim 18 (depends on 15)

The method of claim 15, wherein the first interface size is smaller than the second interface size, such that, the interface deviation operation comprises dynamically shifting the virtual space input interface, such that, a target input portion of the virtual space input interface overlaps with the physical mixed-input input device interface; or
wherein the first interface size is larger than the second interface size, such that, the interface deviation operation comprises receiving the real input based on the physical mixed-input device interface being dynamically shifted, such that, a target input portion of the virtual space input interface overlaps with the physical mixed-input input device interface.

### Claim 19 (depends on 15)

The method of claim 15, further comprising detecting proximity or motion data of external objects to the physical mixed-input device to predict an intent of the user to supplement generating the interaction context.

### Claim 20 (depends on 15)

The method of claim 15, wherein different combination of real input and virtual input trigger corresponding interaction contexts selected from one of the following:
a first interaction context comprising rendering a virtual representation of the physical mixed-input device coupled to a virtual second hand corresponding to a physical second hand coupled to the physical mixed-input device to receive input on the physical mixed-input device;
a second interaction context comprising rendering the virtual representation of the physical mixed-input device coupled to the virtual second hand and a distant virtual first hand corresponding to a physical first hand to receive input on the physical mixed-input device;
a third interaction context comprising rendering a virtual representation of the physical mixed-input device interface on the virtual representation of the physical mixed-input device on the virtual second hand corresponding to the physical second hand to receive input on the physical mixed-input device; and
a fourth interaction context comprising rendering the virtual representation of the physical mixed-input device coupled to the virtual second hand and a floating virtual space input interface and a floating virtual first hand corresponding to the physical first hand to receive input on the physical mixed-input device.
