# US 2018/0321754 A1 · Remote control of a desktop application via a mobile device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0321754 A1
- Document ID: US20180321754A1
- Status: Published application
- Kind: A1
- Application number: 16/035,335
- Filed: 2018-07-13
- Published: 2018-11-08
- Earliest claimed: 2013-11-05
- Assignees: Intuit Inc.
- Applicants: Intuit Inc.
- Inventors: Christopher Z. LESNER
- Web page: https://patentgrep.com/patent/US20180321754A1

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/048: Interaction techniques based on graphical user interfaces [GUI]
- H04L 67/025: for remote control or remote monitoring of applications
- H04L 67/131: Protocols for games, networked simulations or virtual reality
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

One embodiment of the present invention provides a system for using a mobile device to remotely control a desktop application that was configured for use with a pointing device. During operation, the system receives at a mobile device, from a user, a connection request to connect to a desktop application executing on a remote device. In response to the connection request, the system establishes a remote control session between the mobile device and the remote device. Next, the system receives, at the mobile device, a video stream from a first camera coupled to the mobile device. The system then analyzes the video stream, at the mobile device, to identify a gesture made by the user. Next, the system identifies, at the mobile device, a pointing device command corresponding to the gesture. Finally, the system sends the pointing device command to the desktop application executing on the remote device.

## Claims

### Claim 1

A mobile device configured to control a remote device, comprising:
a memory comprising executable instructions;
a chip configured for determining motion vectors;
a touchscreen display;
a network interface;
a first camera coupled to an opposite side of the mobile device from the touch screen display of the mobile device;
a second camera;
a processor in data communication with the memory, the processor configured to execute the executable instructions and cause the mobile device to perform a method for controlling a remote device, the method comprising:
establishing, via the network interface, a remote control session between the mobile device and a remote device;
receiving, via the first camera, a first video stream;
receiving, via the touch screen display, a selection of an area corresponding to a first object within the first video stream displayed on the touch screen display;
analyzing, by the chip configured for determining motion vectors, the first video stream in order to identify a first gesture made by the first object within the first video stream;
identifying, by the processor, a pointing device command corresponding to the first gesture;
sending, from the mobile device to the remote device via the network interface, the pointing device command, the pointing device command being configured to cause a cursor to move within an application executing on the remote device;
receiving, via the second camera, a second video stream;
analyzing, by the chip configured for determining motion vectors, the second video stream to determine a second gesture corresponding to motion of a second object within the second video stream;
identifying, by the processor, a selection command corresponding to the second gesture; and
sending, from the mobile device to the remote device via the network interface, the selection command.

### Claim 2 (depends on 1)

The mobile device of claim 1, further comprising:
an accelerometer,
wherein the method further comprises: compensating for motion of the mobile device while receiving the first video stream based on acceleration data from the accelerometer.

### Claim 3 (depends on 1)

The mobile device of claim 1, wherein the second camera is coupled to the opposite side of the mobile device from the touch screen display of the mobile device.

### Claim 4 (depends on 1)

The mobile device of claim 1, wherein the second camera is coupled to a side of the mobile device comprising the touch screen display.

### Claim 5 (depends on 4)

The mobile device of claim 4, wherein the second object is one of an eye or an eyelid of a user operating the mobile device.

### Claim 6 (depends on 1)

The mobile device of claim 1, wherein the remote control session utilizes a Virtual Network Computing (VNC) protocol.

### Claim 7 (depends on 1)

The mobile device of claim 1, wherein the mobile device is a smartphone.

### Claim 8

A method for using a mobile device to control a remote device:
establishing, at a mobile device, a remote control session between the mobile device and a remote device;
receiving, at the mobile device, a first video stream from a first camera coupled to the mobile device, the first camera being coupled to an opposite side of the mobile device from a touch screen display of the mobile device;
receiving, via the touch screen display, a selection of an area corresponding to a first object within the first video stream displayed on the touch screen display;
analyzing, at the mobile device using hardware configured for determining motion vectors, the first video stream in order to identify a first gesture made by the first object within the first video stream;
identifying, at the mobile device, a pointing device command corresponding to the first gesture;
sending, from the mobile device to the remote device, the pointing device command, the pointing device command being configured to cause a cursor to move within an application executing on the remote device;
receiving, at the mobile device, a second video stream from a second camera coupled to the mobile device;
analyzing, at the mobile device using the hardware configured for determining motion vectors, the second video stream to determine a second gesture corresponding to motion of a second object within the second video stream;
identifying, at the mobile device, a selection command corresponding to the second gesture; and
sending, from the mobile device to the remote device, the selection command.

### Claim 9 (depends on 8)

The method of claim 8, further comprising: using an accelerometer coupled to the mobile device to compensate for motion of the mobile device while receiving the first video stream.

### Claim 10 (depends on 8)

The method of claim 8, wherein the second camera is coupled to the opposite side of the mobile device from the touch screen display of the mobile device.

### Claim 11 (depends on 8)

The method of claim 8, wherein the second camera is coupled to a side of the mobile device comprising the touch screen display.

### Claim 12 (depends on 11)

The method of claim 11, wherein the second object is one of an eye or an eyelid of a user operating the mobile device.

### Claim 13 (depends on 8)

The method of claim 8, wherein the remote control session utilizes a Virtual Network Computing (VNC) protocol.

### Claim 14 (depends on 8)

The method of claim 8, wherein the mobile device is a smartphone.

### Claim 15

A non-transitory computer-readable medium comprising executable instructions that when executed by a mobile device, cause the mobile device to perform a method for using the mobile device to control a remote device, the method comprising:
establishing, at the mobile device, a remote control session between the mobile device and a remote device;
receiving, at the mobile device, a first video stream from a first camera coupled to the mobile device, the first camera being coupled to an opposite side of the mobile device from a touch screen display of the mobile device;
receiving, via the touch screen display, a selection of an area corresponding to a first object within the first video stream displayed on the touch screen display;
analyzing, at the mobile device using hardware configured for determining motion vectors, the first video stream in order to identify a first gesture made by the first object within the first video stream;
identifying, at the mobile device, a pointing device command corresponding to the first gesture;
sending, from the mobile device to the remote device, the pointing device command, the pointing device command being configured to cause a cursor to move within an application executing on the remote device;
receiving, at the mobile device, a second video stream from a second camera coupled to the mobile device;
analyzing, at the mobile device using the hardware configured for determining motion vectors, the second video stream to determine a second gesture corresponding to motion of a second object within the second video stream;
identifying, at the mobile device, a selection command corresponding to the second gesture; and
sending, from the mobile device to the remote device, the selection command.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the method further comprises: using an accelerometer coupled to the mobile device to compensate for motion of the mobile device while receiving the first video stream.

### Claim 17 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the second camera is coupled to the opposite side of the mobile device from the touch screen display of the mobile device.

### Claim 18 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the second camera is coupled to a side of the mobile device comprising the touch screen display.

### Claim 19 (depends on 18)

The non-transitory computer-readable medium of claim 18, wherein the second object is one of an eye or an eyelid of a user operating the mobile device.

### Claim 20 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the mobile device is a smartphone.
