# US 10,120,474 B2 · Touch screen testing platform for engaging a dynamically positioned target feature

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,120,474 B2
- Document ID: US10120474B2
- Status: Granted
- Kind: B2
- Application number: 15/208,536
- Filed: 2016-07-12
- Published: 2016-11-03
- Granted: 2018-11-06
- Earliest claimed: 2010-12-09
- Assignees: T-Mobile USA, Inc.
- Applicants: T-Mobile USA, Inc.
- Inventors: David Ross Jenkinson
- Web page: https://patentgrep.com/patent/US10120474B2

## CPC

- B25J 11/00: Manipulators not otherwise provided for
- B25J 9/02: characterised by movement of the arms, e.g. cartesian coordinate type (B25J9/06 takes precedence)
- G06F 3/0412: {Digitisers structurally integrated in a display}
- G06F 11/2221: {to test input/output devices or peripheral units}
- G01R 1/06705: {Apparatus for holding or moving single probes (for moving multiple probe heads or ICs under test G01R31/2886)}
- Y10S 901/47: Optical
- Y10T 74/20305: Robotic arm
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G01R: MEASURING ELECTRIC VARIABLES; MEASURING MAGNETIC VARIABLES (indicating correct tuning of resonant circuits H03J3/12)
- Y10S: TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10T: TECHNICAL SUBJECTS COVERED BY FORMER US CLASSIFICATION

## Abstract

A touch screen testing platform may be used to engage a dynamically positioned target feature being displayed on a touch screen enabled device during a testing protocol. The platform may record imagery displayed by the touch screen device and then analyze the imagery to locate the target feature within a reference coordinate system. The platform may recognize that the target feature is missing from the imagery and respond by causing the touch screen device to scroll through a command menu and/or toggle through virtual screens. Once located, the platform may instruct a robotic device tester to select the target feature by contacting the touch screen at the identified location using a conductive tip designed to simulate a user's fingertip. Prior to running a test, the camera may be focused to a point that is offset from the display screen of the touch screen device.

## Claims

### Claim 1

A method comprising:
determining a plurality of instructions to cause a robot to move a tip to selectively engage a touch screen of a touch screen device to provide the touch screen device with inputs corresponding to a testing protocol, wherein the touch screen is within a field-of-view of a camera, and wherein the inputs simulate user interactions with the touch screen device;
executing a first instruction, of the plurality of instructions, to cause the robot to provide the touch screen device with a first input, the first input causing the touch screen device to display imagery that includes a command menu that includes at least a target feature, a location of the target feature depending at least partially on a state of the command menu;
receiving, from the camera, a recording of the imagery that includes at least the target feature;
identifying, based on the recording of the imagery, the location of the target feature within the field-of-view of the camera;
based on the identifying the location of the target feature, executing a second instruction to cause the robot to move the tip to selectively engage the touch screen at the location of the target feature thereby providing the touch screen device with a second input that is associated with the target feature; and
wherein the touch screen device is configured to perform a lodging task to generate a data log during the testing protocol, and wherein the second input causes the touch screen device to initiate, pause, resume, or terminate the logging task.

### Claim 2 (depends on 1)

The method as recited in claim 1, further comprising:
determining, based on the recording of the imagery, an absence of the target feature from the field-of-view of the camera; and
in response to the determining the absence of the target feature, executing a third instruction to cause the robot to provide the touch screen device with a third input, the third input causing the touch screen device to scroll through the command menu thereby causing the imagery to include the target feature.

### Claim 3 (depends on 1)

The method as recited in claim 1, further comprising:
prior to the executing the first instruction, causing a graphical user interface of a user terminal to display an image of the target feature; and
receiving, at the user terminal, a user input that defines one or more boundaries of the target feature within the image on the graphical user interface, wherein the identifying the location of the target feature within the field-of-view of the camera is based at least on the user input.

### Claim 4 (depends on 1)

The method as recited in claim 1, further comprising receiving a data file corresponding to one or more visual characteristics of the target feature, wherein the identifying the location of the target feature within the field-of-view of the camera is based at least on the data file.

### Claim 5 (depends on 1)

The method as recited in claim 1, wherein a first set of the plurality of instructions correspond to static input locations that are determinable prior to providing the touch screen device with individual instructions of the first set, and wherein a second set of the plurality of instructions correspond to dynamic input locations that are undeterminable prior to providing the touch screen device with individual instructions of the second set.

### Claim 6

A system for engaging a target feature that is dynamically positioned during a testing protocol, the system comprising:
a robot to provide inputs to a touch screen of a touch screen device by moving a tip into contact with the touch screen, wherein the touch screen is within a field-of-view of a camera configured to record imagery rendered by the touch screen device; and
a controller to control movement of the tip by the robot by performing operations of:
providing the robot with a first instruction to cause the robot to provide the touch screen device with a first input;
receiving, from the camera, a recording of the imagery rendered by the touch screen device;
scanning the recording of the imagery to determine a reference coordinate location of the target feature within a reference coordinate system associated with the robot;
determining, based on the scanning, an absence of the target feature from the field-of-view of the camera;
in response to the determining the absence of the target feature, executing a second instruction to cause the robot to provide the touch screen device with a second input thereby causing the imagery to include the target feature; and
based on the reference coordinate location, providing the robot with a third instruction to cause the robot to provide the touch screen device with a third input, the third input to simulate a user selection of the target feature.

### Claim 7 (depends on 6)

The system of claim 6, the operations further comprising:
receiving, via a user terminal, user input associated with boundaries corresponding to a plurality of target features within one or more images on a graphical user interface; and
generating, based on the user input associated with the boundaries, target feature data corresponding to visual characteristics of individual target features of the plurality of target features, wherein the scanning includes accessing a portion of the target feature data associated with the target feature.

### Claim 8 (depends on 6)

The system of claim 6, wherein the third input causes the touch screen device to scroll from a first set of application icons to a second set of application icons, the second set of application icons including the target feature.

### Claim 9 (depends on 6)

The system of claim 6, wherein the first input includes at least one of a tap movement to engage the tip to a single location on the touch screen or a slide movement to drag the tip across a portion of the touch screen.

### Claim 10 (depends on 9)

The system of claim 9, wherein at least one of the tap movement or the slide movement causes the touch screen device to display the target feature.

### Claim 11 (depends on 6)

The system of claim 6, the operations further comprising:
determining, based on the scanning, pixel location data corresponding to at least one pixel of the recording at which the target feature is located; and
determining the reference coordinate location based on the pixel location data.

### Claim 12 (depends on 6)

The system of claim 6, wherein the scanning the recording of the imagery to determine the reference coordinate location includes matching visual characteristics of a predetermined pattern to a portion of the recording of the imagery.

### Claim 13 (depends on 6)

The system of claim 6, wherein the operations further comprise performing a logging task to generate a data log during the testing protocol, and wherein the second input causes the touch screen device to initiate, pause, resume, or terminate the logging task.

### Claim 14

A method comprising:
placing a touch screen device at a testing location that is within a field-of-view of a camera and adjacent to a robot configured to move a tip into contact with a touch screen of the touch screen device;
placing a focal target above the touch screen device;
focusing the camera onto the focal target to cause a focal point of the camera to be in front of or behind the touch screen; and
initiating a computer-implemented testing protocol to test one or more functionalities of the touch screen device, the computer-implemented testing protocol to perform operations including:
receiving, from the camera while the focal point is in front of or behind the touch screen, imagery displayed by the touch screen, the imagery including at least a target feature;
identifying, based on the imagery, a location of the target feature on the touch screen; and
causing the robot to select the target feature by moving the tip into contact with the touch screen at the location.

### Claim 15 (depends on 14)

The method of claim 14, wherein the causing the focal point of the camera to be in front of or behind the touch screen includes:
placing a focal jig at the testing location, the focal jig including at least one focal target;
focusing the camera onto the at least one focal target; and
prior to the placing the touch screen device at the testing location, removing the focal jig from the testing location.

### Claim 16 (depends on 15)

The method of claim 15, wherein the placing a touch screen device at the testing location includes securing the touch screen device to a device fixture, the touch screen being accessible when the touch screen device is in the device fixture.

### Claim 17 (depends on 16)

The method of claim 16, wherein the testing location includes one or more positioning elements configured to commonly mate with each of the focal jig and the device fixture.

### Claim 18 (depends on 14)

The method of claim 14, further comprising performing a logging task to generate a data log during the computer-implemented testing protocol.

### Claim 19 (depends on 14)

The method of claim 14, wherein identifying the location of the target feature on the touch screen comprises determining pixel location data corresponding to at least one pixel at which the target feature is located.