# US 2018/0321755 A1 · Remote control of a desktop application via a mobile device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0321755 A1
- Document ID: US20180321755A1
- Status: Published application
- Kind: A1
- Application number: 16/035,347
- Filed: 2018-07-13
- Published: 2018-11-08
- Earliest claimed: 2013-11-05
- Assignees: Intuit Inc.
- Applicants: Intuit Inc.
- Inventors: Christopher Z. LESNER
- Web page: https://patentgrep.com/patent/US20180321755A1

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/048: Interaction techniques based on graphical user interfaces [GUI]
- H04L 67/025: for remote control or remote monitoring of applications
- H04L 67/131: Protocols for games, networked simulations or virtual reality
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

One embodiment of the present invention provides a system for using a mobile device to remotely control a desktop application that was configured for use with a pointing device. During operation, the system receives at a mobile device, from a user, a connection request to connect to a desktop application executing on a remote device. In response to the connection request, the system establishes a remote control session between the mobile device and the remote device. Next, the system receives, at the mobile device, a video stream from a first camera coupled to the mobile device. The system then analyzes the video stream, at the mobile device, to identify a gesture made by the user. Next, the system identifies, at the mobile device, a pointing device command corresponding to the gesture. Finally, the system sends the pointing device command to the desktop application executing on the remote device.

## Claims

### Claim 1

A wearable device configured to control a remote device, comprising:
a memory comprising executable instructions;
a chip configured for determining motion vectors;
a screen;
a network interface;
a first camera coupled to an opposite side of the wearable device from the screen;
a processor in data communication with the memory, the processor configured to execute the executable instructions and cause the wearable device to perform a method for controlling a remote device, the method comprising:
establishing, via the network interface, a remote control session between the wearable device and a remote device;
receiving, via the first camera, a first video stream;
analyzing, by the chip configured for determining motion vectors, the first video stream in order to identify a first gesture made by a first object within the first video stream;
identifying, by the processor, a pointing device command corresponding to the first gesture; and
sending, from the wearable device to the remote device via the network interface, the pointing device command, the pointing device command being configured to cause a cursor to move within an application executing on the remote device.

### Claim 2 (depends on 1)

The wearable device of claim 1, further comprising:
an accelerometer,
wherein the method further comprises: compensating for motion of the wearable device while receiving the first video stream based on acceleration data from the accelerometer.

### Claim 3 (depends on 1)

The wearable device of claim 1, wherein the method further comprises:
receiving, via a second camera, a second video stream;
analyzing, by the chip configured for determining motion vectors, the second video stream to determine a second gesture corresponding to motion of a second object within the second video stream;
identifying, by the processor, a selection command corresponding to the second gesture; and
sending, from the wearable device to the remote device via the network interface, the selection command.

### Claim 4 (depends on 3)

The wearable device of claim 3, wherein the second camera is coupled to the opposite side of the wearable device from the screen of the wearable device.

### Claim 5 (depends on 3)

The wearable device of claim 3, wherein the second camera is coupled to a side of the wearable device comprising the screen.

### Claim 6 (depends on 5)

The wearable device of claim 5, wherein the second object is one of an eye or an eyelid of a user operating the wearable device.

### Claim 7 (depends on 1)

The wearable device of claim 1, wherein the wearable device is a pair of glasses, and wherein the screen is embedded in the pair of glasses.

### Claim 8

A method for using a wearable device to control a remote device:
establishing, at a wearable device, a remote control session between the wearable device and a remote device;
receiving, at the wearable device, a first video stream from a first camera coupled to the wearable device, the first camera being coupled to an opposite side of the wearable device from a screen of the wearable device;
analyzing, at the wearable device using hardware configured for determining motion vectors, the first video stream in order to identify a first gesture made by a first object within the first video stream;
identifying, at the wearable device, a pointing device command corresponding to the first gesture; and
sending, from the wearable device to the remote device, the pointing device command, the pointing device command being configured to cause a cursor to move within an application executing on the remote device.

### Claim 9 (depends on 8)

The method of claim 8, further comprising: using an accelerometer coupled to the wearable device to compensate for motion of the wearable device while receiving the first video stream.

### Claim 10 (depends on 8)

The method of claim 8, further comprising:
receiving, at the wearable device, a second video stream from a second camera coupled to the wearable device;
analyzing, at the wearable device using the hardware configured for determining motion vectors, the second video stream to determine a second gesture corresponding to motion of a second object within the second video stream;
identifying, at the wearable device, a selection command corresponding to the second gesture; and
sending, from the wearable device to the remote device, the selection command.

### Claim 11 (depends on 10)

The method of claim 10, wherein the second camera is coupled to the opposite side of the wearable device from the screen of the wearable device.

### Claim 12 (depends on 10)

The method of claim 10, wherein the second camera is coupled to a side of the wearable device comprising the screen.

### Claim 13 (depends on 12)

The method of claim 12, wherein the second object is one of an eye or an eyelid of a user operating the wearable device.

### Claim 14 (depends on 8)

The method of claim 8, wherein the wearable device is a pair of glasses, and wherein the screen is embedded in the pair of glasses.

### Claim 15

A non-transitory computer-readable medium comprising executable instructions that when executed by a processor within a wearable device, cause the wearable device to perform a method for using the wearable device to control a remote device, the method comprising:
establishing, at the wearable device, a remote control session between the wearable device and a remote device;
receiving, at the wearable device, a first video stream from a first camera coupled to the wearable device, the first camera being coupled to an opposite side of the wearable device from a screen of the wearable device;
analyzing, at the wearable device using hardware configured for determining motion vectors, the first video stream in order to identify a first gesture made by a first object within the first video stream;
identifying, at the wearable device, a pointing device command corresponding to the first gesture; and
sending, from the wearable device to the remote device, the pointing device command, the pointing device command being configured to cause a cursor to move within an application executing on the remote device.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the method further comprises: using an accelerometer coupled to the wearable device to compensate for motion of the wearable device while receiving the first video stream.

### Claim 17 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the method further comprises:
receiving, at the wearable device, a second video stream from a second camera coupled to the wearable device;
analyzing, at the wearable device using the hardware configured for determining motion vectors, the second video stream to determine a second gesture corresponding to motion of a second object within the second video stream;
identifying, at the wearable device, a selection command corresponding to the second gesture; and
sending, from the wearable device to the remote device, the selection command.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein the second camera is coupled to a side of the wearable device comprising the screen.

### Claim 19 (depends on 18)

The non-transitory computer-readable medium of claim 18, wherein the second object is one of an eye or an eyelid of a user operating the wearable device.

### Claim 20 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the wearable device is a pair of glasses, and wherein the screen is embedded in the pair of glasses.
