# US 2014/0208170 A1 · Method for monitoring a graphical user interface on a second computer display from a first computer

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2014/0208170 A1
- Document ID: US20140208170A1
- Status: Published application
- Kind: A1
- Application number: 14/219,968
- Filed: 2014-03-19
- Published: 2014-07-24
- Earliest claimed: 2003-10-01
- Assignees: Testplant, Inc.
- Applicants: Testplant, Inc.
- Inventors: Douglas P. SIMONS; Jonathan D. GILLASPIE
- Web page: https://patentgrep.com/patent/US20140208170A1

## CPC

- G06F 11/2273: {Test methods}
- G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F 3/04817: using icons (graphical or visual programming using iconic symbols G06F8/34)
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/0487: using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F 9/452: {Remote windowing, e.g. X-Window System, desktop virtualisation (protocols for virtual reality H04L67/131)}
- G06F 11/3688: {for test execution, e.g. scheduling of test suites}
- G06F 11/3698: Environments for analysis, debugging or testing of software
- G03G 15/50: {Machine control of apparatus for electrographic processes using a charge pattern, e.g. regulating differents parts of the machine, multimode copiers, microprocessor control (sequencing control G03G21/14)}
- G06F 3/1415: {with means for detecting differences between the image stored in the host and the images displayed on the displays}
- G06F 11/3414: {Workload generation, e.g. scripts, playback}
- G06F 11/36: Prevention of errors by analysis, debugging or testing of software
- G06F 11/3672: {Test management}
- G06F 2201/875: Monitoring of systems including the internet
- G06T 7/0008: {checking presence/absence}
- G06T 7/001: {using an image reference approach}
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 9/004: {Predictors, e.g. intraframe, interframe coding}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G03G: ELECTROGRAPHY; ELECTROPHOTOGRAPHY; MAGNETOGRAPHY
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A high level interface between a remote computer and local computer operator permits the remote computer to be controlled via a sequence of interactions. The remote computer may be monitored for display information which is expected, and also controlled in the event the expected information either is or is not obtained. Command language extensions are provided which extend and complement a basic scripting language. Scripts with embedded command language extensions may be executed through a remote interface, permitting remote testing, operation and evaluation. The development of the scripts, including embedded command language extensions, may be prepared through execution of commands at the local computer on a depiction of the GUI received from the remote computer, and subsequently saved as a script.

## Claims

### Claim 1

A computer system for testing a mobile device, the computer system being connected to the mobile device and comprising:
a memory unit that stores instructions; and
at least one processing unit that executes the instructions to:
receive a representation of a graphical user interface of the mobile device;
detect a presence or absence of a first graphical element in the received representation of the mobile device graphical user interface based on an image analysis; and
based on a result of detecting a presence of the first graphical element, automatically provide a signal to the mobile device to initiate an input action on the mobile device graphical user interface in accordance with a testing script.

### Claim 2 (depends on 1)

The computer system of claim 1, wherein the input action emulates an input command from a peripheral input device including at least one selected from a group comprising: a keyboard, a mouse, and a touch screen device.

### Claim 3 (depends on 1)

The computer system of claim 1, wherein the input action includes at least one selected from a group comprising: a mouse click, a mouse movement, and text entry.

### Claim 4 (depends on 1)

The computer system of claim 1, wherein the first graphical element comprises less than the entire representation of the mobile device system graphical user interface.

### Claim 5 (depends on 4)

The computer system of claim 4, wherein the first graphical element comprises at least one selected from a group comprising: an icon, text, a text box, and a prompt.

### Claim 6 (depends on 1)

The computer system of claim 1, wherein the at least one processing unit further executes the instructions to:
automatically provide as the signal, a first signal to execute a first input action at the mobile device, in response to detecting a presence of the first graphical element in the received representation of the mobile device graphical user interface during the search.

### Claim 7 (depends on 1)

The computer system of claim 1, wherein the at least one processing unit further executes the instructions to:
indicate a failure, in response to detecting an absence of the first graphical element in the received representation of the mobile device graphical user interface during the search.

### Claim 8 (depends on 1)

The computer system of claim 1, wherein the at least one processing unit further executes the instructions to:
receive an additional representation of the graphical user interface of the mobile device subsequent to automatically providing the signal to the mobile device to initiate the input action in accordance with the testing script; and
detect a presence or absence of a second graphical element in the additionally received representation of the mobile device graphical user interface based on an image analysis.

### Claim 9 (depends on 8)

The computer system of claim 8, wherein the at least one processing unit further executes the instructions to:
based on a result of detecting a presence of the second graphical element, automatically provide a signal to the mobile device to initiate another input action on the mobile device graphical user interface in accordance with the testing script.

### Claim 10 (depends on 1)

The computer system of claim 1, further comprising:
a local display that depicts the received representation of the mobile device graphical user interface.

### Claim at position 11 (depends on 1)

11 The computer system of claim 1, wherein the at least one processing unit further executes the instructions to:
capture user input effected with respect to the representation of the mobile device graphical user interface display; and
generate the testing script based on the captured user input.

### Claim 12

A method implemented at a computer system for testing a mobile device connected to the computer system, the method executed by at least one processor and comprising:
receiving a representation of a graphical user interface of the mobile device;
detecting a presence or absence of a first graphical element in the received representation of the mobile device graphical user interface based on an image analysis; and
based on a result of detecting a presence of the first graphical element, automatically providing a signal to the mobile device to initiate an input action on the mobile device graphical user interface in accordance with a testing script.

### Claim 13 (depends on 12)

The method of claim 12, wherein the input action emulates an input command from a peripheral input device including at least one selected from a group comprising: a keyboard, a mouse, and a touch screen device.

### Claim 14 (depends on 12)

The method of claim 12, wherein the input action includes at least one selected from a group comprising: a mouse click, a mouse movement, and text entry.

### Claim 15 (depends on 12)

The method of claim 12, wherein the first graphical element comprises less than the entire representation of the mobile device system graphical user interface.

### Claim 16 (depends on 14)

The method of claim 14, wherein the first graphical element comprises at least one selected from a group comprising: an icon, text, a text box, and a prompt.

### Claim 17 (depends on 12)

The method of claim 12, wherein the method further comprises:
automatically providing as the signal, a first signal to execute a first input action at the mobile device, in response to detecting a presence of the first graphical element in the received representation of the mobile device graphical user interface during the search.

### Claim 18 (depends on 12)

The method of claim 12, further comprising:
indicating a failure, in response to detecting an absence of the first graphical element in the received representation of the mobile device graphical user interface during the search.

### Claim 19 (depends on 12)

The method of claim 12, further comprising:
receiving an additional representation of the graphical user interface of the mobile device subsequent to automatically providing the signal to the mobile device to initiate the input action in accordance with the testing script; and
detecting a presence or absence of a second graphical element in the additionally received representation of the mobile device graphical user interface based on an image analysis.

### Claim 20 (depends on 19)

The method of claim 19, further comprising:
based on a result of detecting a presence of the second graphical element, automatically providing a signal to the mobile device to initiate another input action on the mobile device graphical user interface in accordance with the testing script.

### Claim 21 (depends on 12)

The method of claim 12, further comprising:
depicting, on a local display, the received representation of the mobile device graphical user interface.

### Claim 22 (depends on 12)

The method of claim 12, further comprising:
capturing user input effected with respect to the representation of the mobile device graphical user interface display; and
generating the testing script based on the captured user input.

### Claim 23

A non-transitory computer-readable storage medium storing instructions for enabling a computer system to implement a method for testing a mobile device connected to the computer system, the method comprising:
receiving a representation of a graphical user interface of the mobile device;
detecting a presence or absence of a first graphical element in the received representation of the mobile device graphical user interface based on an image analysis; and
based on a result of detecting a presence of the first graphical element, automatically providing a signal to the mobile device to initiate an input action on the mobile device graphical user interface in accordance with a testing script.

### Claim 24 (depends on 23)

The non-transitory computer-readable storage medium of claim 23, wherein the input action emulates an input command from a peripheral input device including at least one selected from a group comprising: a keyboard, a mouse, and a touch screen device.

### Claim 25 (depends on 23)

The non-transitory computer-readable storage medium of claim 23, wherein the input action includes at least one selected from a group comprising: a mouse click, a mouse movement, and text entry.

### Claim 26 (depends on 23)

The non-transitory computer-readable storage medium of claim 23, wherein the first graphical element comprises less than the entire representation of the mobile device system graphical user interface.

### Claim 27 (depends on 26)

The non-transitory computer-readable storage medium of claim 26, wherein the first graphical element comprises at least one selected from a group comprising: an icon, text, a text box, and a prompt.

### Claim 28 (depends on 23)

The non-transitory computer-readable storage medium of claim 23, wherein the method further comprises:
automatically providing as the signal, a first signal to execute a first input action at the mobile device, in response to detecting a presence of the first graphical element in the received representation of the mobile device graphical user interface during the search.

### Claim 29 (depends on 12)

The non-transitory computer-readable storage medium of claim 12, wherein the method further comprises:
indicating a failure, in response to detecting an absence of the first graphical element in the received representation of the mobile device graphical user interface during the search.

### Claim 30 (depends on 23)

The non-transitory computer-readable storage medium of claim 23, wherein the method further comprises:
receiving an additional representation of the graphical user interface of the mobile device subsequent to automatically providing the signal to the mobile device to initiate the input action in accordance with the testing script; and
detecting a presence or absence of a second graphical element in the additionally received representation of the mobile device graphical user interface based on an image analysis.
