# US 10,484,437 B2 · Remote support service with two-way smart whiteboard

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,484,437 B2
- Document ID: US10484437B2
- Status: Granted
- Kind: B2
- Application number: 15/607,707
- Filed: 2017-05-30
- Published: 2017-09-14
- Granted: 2019-11-19
- Earliest claimed: 2015-01-21
- Assignees: LogMeIn, Inc.
- Applicants: LogMeIn, Inc.
- Inventors: Sándor Balázs Bányai; András Sulykos; Viktor Bernardt
- Web page: https://patentgrep.com/patent/US10484437B2

## CPC

- H04L 65/4015: {where at least one of the additional parallel sessions is real time or time sensitive, e.g. white board sharing, collaboration or spawning of a subconference}
- G06F 1/3209: Monitoring remote activity, e.g. over telephone lines or network connections
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 3/1423: {controlling a plurality of local displays, e.g. CRT and flat panel display}
- G06F 3/1454: {involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay}
- G06F 9/452: {Remote windowing, e.g. X-Window System, desktop virtualisation (protocols for virtual reality H04L67/131)}
- G06F 9/453: {Help systems}
- G06F 21/602: {Providing cryptographic facilities or services}
- G06Q 10/10: Office automation; Time management
- H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46)
- H04L 67/131: Protocols for games, networked simulations or virtual reality
- H04M 1/72415: for remote control of appliances
- H04N 21/42204: {User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor (constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205)}
- H04L 67/125: involving control of end-device applications over a network
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Mobile devices, such as a smartphone, tablet or wearable, are leveraged to facilitate remote support between a user of the device, and a remote technician. In this approach, the mobile device is used as a tool on behalf of the remote technician. In particular, and in one embodiment, the mobile device is used to transfer images to the remote technician's desktop. The remote technician desktop is provisioned with a whiteboard functionality by which the remote technical draws on the camera image that the remote technician is then viewing. The resulting annotation(s) are then transferred back over the network and drawn on the end user's camera view. In this manner, the technician highlights any area of the camera image, and those highlights are visible on the end user mobile device regardless of whether the local end user rotates the view or moves the mobile device camera during the support session. In an alternative embodiment, collaborative support is provided between or among the user and one or more technicians using a two-way, bi-directional smart whiteboard.

## Claims

### Claim 1

A method of remote support executed during a support session established between a first computing entity and one or more second computing entities that are coupled to the first computing entity, the method carried out at one of the second computing entities as an image feed generated at the first computing entity is being shared with the one or more second computing entities, comprising:
generating and broadcasting sets of movement computations from the given second computing entity to the first computing entity and any other second computing entity, wherein the sets of movement computations represent transformation data to be applied to maintain position and orientation of an annotation relative to the image feed;
receiving one or more sets of movement computations output from the first computing entity and any other second computing entity;
determining, based on the sets of movement computations broadcast and received, whether the given second entity should continue to generate and broadcast sets of movement computations; and
when it is determined that the given second entity should continue to generate and broadcast sets of movement computations, having the second computing entity assume and maintain responsibility for generating one or more additional sets of movement computations.

### Claim 2 (depends on 1)

The method as described in claim 1 wherein the determining operation is based on a count of sets of movement computations sent from the given second computing entity, and a count of the sets of movement computations received by the given second computing entity.

### Claim 3 (depends on 2)

The method as described in claim 2 wherein the given second entity assumes and maintains responsibility for generating the one or more additional sets of movement computations when the counts reflect that the given second entity has a largest number of movement computations sent.

### Claim 4 (depends on 2)

The method as described in claim 2 wherein the determining operation is a function of one or more of: an operational state of the second computing entity.

### Claim 5 (depends on 1)

The method as described in claim 1 wherein a set of movement computations represents transformation data for a tracked area on the image feed.

### Claim 6 (depends on 5)

The method as described in claim 5 wherein the transformation data is a transformation matrix having a set of elements whose values represent an action that is one of: a move operation, a scaling operation, a rotate around origin operation, and combinations thereof.

### Claim 7 (depends on 1)

The method as described in claim 1 wherein, when it is determined that the given second entity should not continue to generate and broadcast sets of movement computations, having the second computing entity cease generating one or more additional sets of movement computations.

### Claim 8 (depends on 1)

The method as described in claim 1 wherein, when it is determined that the given second entity should not continue to generate and broadcast sets of movement computations, having the second computing entity continue generating one or more additional sets of movement computations.

### Claim 9 (depends on 4)

The method as described in claim 4 wherein the operational state of the second computing entity is one of: a battery level, an indication that the entity is receiving power from a line source, and a number or type of application then running on the entity.

### Claim 10 (depends on 1)

The method as described in claim 1 wherein the determining operation is carried out periodically.

### Claim 11 (depends on 1)

The method as described in claim 1 wherein the determining operation is carried out upon a given occurrence.

### Claim 12 (depends on 10)

The method as described in claim 10 wherein the given occurrence is a determination that frame rate of the image feed falls below a given threshold.

### Claim 13 (depends on 1)

The method as described in claim 1 wherein the first computing entity is one of: a smartphone, a tablet, and a wearable.

### Claim 14 (depends on 13)

The method as described in claim 13 wherein the second computing entity is a mobile device.

### Claim 15 (depends on 1)

The method as described in claim 1 wherein the remote support session is implemented over a peer-to-peer communication channel.

### Claim 16

A mobile device apparatus having a camera, comprising:
a processor;
computer memory holding computer program instructions executed by the processor during a remote support session established with a support computing entity, the computer program instructions operative to:
render on a display interface image frames together with an annotation, wherein a position and orientation of the annotation is determined by transformation data; and
a control routine executed during a benchmarking phase to determine responsibility for generating the transformation data during the remote support session, wherein the control routine generates and sends sets of movement computations to one or more other computing entities, receives one or more sets of movement computations output from the one or more other computing entities, and determines, based on the sets of movement computations transmitted and received whether the control routine should continue to generate and transmit sets of movement computations, wherein the control routine determines it should continue to generate and transmit sets of movement computations based on one of: a count of transmitted sets, and a count of transmitted sets, together with an operational characteristic of the mobile device apparatus.

### Claim 17 (depends on 16)

The mobile device apparatus as described in claim 16 wherein the benchmarking phase is repeated periodically or as needed during the remote support session.