# US 2023/0409467 A1 · System and Method for User Interface Testing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0409467 A1
- Document ID: US20230409467A1
- Status: Published application
- Kind: A1
- Application number: 18/135,240
- Filed: 2023-04-17
- Published: 2023-12-21
- Earliest claimed: 2022-06-17
- Assignees: Verizon Patent and Licensing Inc.
- Applicants: Verizon Patent and Licensing Inc.
- Inventors: Karthikeyan Kadirvel; Sandeep K Kotha
- Web page: https://patentgrep.com/patent/US20230409467A1

## CPC

- G06F 9/451: Execution arrangements for user interfaces
- G06F 11/3698: Environments for analysis, debugging or testing of software
- G06F 11/3692: {for test results analysis}
- G06F 2201/81: Threshold
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

In an example, images of an interface of an application may be identified. The images may be clustered into groups associated with layouts of the interface. An image of a first group associated with a first layout may be compared with a second image of the first group to determine a first set of regions, of the interface, in which the second image is different than the first image. Based upon the first set of regions, a set of dynamic regions of the interface may be determined. A fourth image of the interface may be identified. The fourth image may be determined to correspond to the first layout of the interface. Based upon the set of dynamic regions, a first portion of the fourth image may be compared with a first portion of the first image to determine user interface testing information associated with the interface of the application.

## Claims

### Claim 1

A method, comprising:
identifying a plurality of images of an interface of an application;
clustering the plurality of images into a plurality of groups that are each associated with a layout of the interface, wherein the plurality of groups comprise:
a first group of images associated with a first layout of the interface; and
a second group of images associated with a second layout of the interface;
comparing a first image of the first group of images with a second image of the first group of images to determine a first set of regions, of the interface, in which the second image is different than the first image;
comparing the first image with a third image of the first group of images to determine a second set of regions, of the interface, in which the third image is different than the first image;
determining, based upon the first set of regions and the second set of regions, a set of dynamic regions of the interface;
identifying a fourth image of the interface;
determining that the fourth image corresponds to the first layout of the interface; and
comparing, based upon the set of dynamic regions, a first portion of the fourth image with a first portion of the first image to determine user interface testing information associated with the interface of the application.

### Claim 2 (depends on 1)

The method of claim 1, comprising:
comparing a fifth image of the second group of images with a sixth image of the second group of images to determine a third set of regions, of the interface, in which the sixth image is different than the fifth image;
comparing the fifth image with a seventh image of the second group of images to determine a fourth set of regions, of the interface, in which the seventh image is different than the fifth image;
determining, based upon the third set of regions and the fourth set of regions, a second set of dynamic regions of the interface;
identifying an eighth image of the interface;
determining that the eighth image corresponds to the second layout of the interface; and
in response to determining that the eighth image corresponds to the second layout of the interface, comparing, based upon the second set of dynamic regions, a first portion of the eighth image with a first portion of the fifth image to determine second user interface testing information associated with the interface of the application.

### Claim 3 (depends on 1)

The method of claim 1, wherein:
clustering the plurality of images into the plurality of groups comprises:
determining, based upon the plurality of images, a plurality of sets of features associated with the plurality of images; and
performing, based upon the plurality of sets of features, a first clustering process to cluster the plurality of images into a second plurality of groups.

### Claim 4 (depends on 3)

The method of claim 3, wherein:
determining a set of features, of the plurality of sets of features, associated with a fifth image comprises:
comparing the fifth image with a sixth image to determine one or more comparison metrics associated with the fifth image and the sixth image; and
determining a histogram associated with the fifth image.

### Claim 5 (depends on 4)

The method of claim 4, wherein:
the one or more comparison metrics comprise a Structural Similarity Index (SSIM) score.

### Claim 6 (depends on 4)

The method of claim 4, wherein:
clustering the plurality of images into the plurality of groups comprises:
generating layout representations of images of a third group of images of the second plurality of groups;
determining one or more layout similarity scores associated with similarities between the layout representations; and
determining whether to cluster the third group of images into sub-groups based upon the one or more layout similarity scores.

### Claim 7 (depends on 6)

The method of claim 6, wherein:
generating a layout representation of the layout representations comprises:
determining edges of an image of the third group of images using canny edge detection;
determining contours of the image based upon the edges;
removing, from the contours, contours with sizes smaller than a threshold size to generate a filtered set of contours; and
generating the layout representation based upon the filtered set of contours.

### Claim 8 (depends on 6)

The method of claim 6, wherein clustering the plurality of images into the plurality of groups comprises:
in response to determining to cluster the third group of images into sub-groups, performing, based upon the one or more layout similarity scores, a second clustering process to cluster the third group of images into two or more groups comprising the first group of images and the second group of images.

### Claim 9 (depends on 6)

The method of claim 6, wherein clustering the plurality of images into the plurality of groups comprises:
in response to determining not to cluster the third group of images into sub-groups, including the third group of images in the plurality of groups without clustering the third group of images into sub-groups.

### Claim 10 (depends on 3)

The method of claim 3, comprising:
determining a first quantity of groups for the first clustering process, wherein the first clustering process is performed, based upon the first quantity of groups, such that a quantity of groups of the second plurality of groups is equal to the first quantity of groups.

### Claim 11 (depends on 10)

The method of claim 10, wherein:
determining the first quantity of groups comprises:
determining, based upon the plurality of sets of features, variance metrics associated with a plurality of quantities of groups; and
selecting the first quantity of groups from the plurality of quantities of groups based upon the variance metrics.

### Claim 12 (depends on 11)

The method of claim 11, wherein selecting the first quantity of groups is performed based upon a determination that a variance metric associated with the first quantity of groups corresponds to an elbow of a curve associated with the variance metrics.

### Claim 13 (depends on 1)

The method of claim 1, wherein:
determining that the fourth image corresponds to the first layout of the interface comprises:
generating a first layout representation of the fourth image;
comparing the first layout representation with layout representations associated with multiple layouts of the interface to determine a plurality of layout similarity scores associated with similarities between the first layout representation and the layout representations associated with the multiple layouts; and
determining that a layout similarity score associated with the first layout at least one of:
exceeds a threshold layout similarity score; or
is the highest layout similarity score of the plurality of layout similarity scores.

### Claim 14 (depends on 13)

The method of claim 13, wherein:
generating the first layout representation comprises:
determining edges of the fourth image using canny edge detection;
determining contours of the fourth image based upon the edges;
removing, from the contours, contours with sizes smaller than a threshold size to generate a filtered set of contours; and
generating the first layout representation based upon the filtered set of contours.

### Claim 15 (depends on 1)

The method of claim 1, wherein:
the set of dynamic regions comprise the first set of regions and the second set of regions.

### Claim 16 (depends on 1)

The method of claim 1, wherein at least one of:
the first portion of the first image corresponds to a set of static regions, of the interface, different than the set of dynamic regions of the interface;
the first portion of the fourth image corresponds to the set of static regions of the interface; or
the user interface testing information is indicative of a region, in the set of static regions of the interface, in which in the fourth image is different than the first image.

### Claim 17

A non-transitory computer-readable medium storing instructions that when executed perform operations comprising:
identifying a plurality of images of an interface of an application;
clustering the plurality of images into a plurality of groups that are each associated with a layout of the interface, wherein the plurality of groups comprise:
a first group of images associated with a first layout of the interface; and
a second group of images associated with a second layout of the interface;
comparing a first image of the first group of images with a second image of the first group of images to determine a first set of regions, of the interface, in which the second image is different than the first image;
comparing the first image with a third image of the first group of images to determine a second set of regions, of the interface, in which the third image is different than the first image;
determining, based upon the first set of regions and the second set of regions, a set of dynamic regions of the interface;
identifying a fourth image of the interface;
determining that the fourth image corresponds to the first layout of the interface; and
comparing, based upon the set of dynamic regions, a first portion of the fourth image with a first portion of the first image to determine user interface testing information associated with the interface of the application.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein:
the user interface testing information indicates that the interface has a user interface error based upon a determination that a similarity score associated with a similarity between the first portion of the first image and the first portion of the fourth image is less than a threshold.

### Claim 19

A device comprising:
a processor configured to execute instructions to perform operations comprising:
identifying a plurality of images of an interface of an application;
clustering the plurality of images into a plurality of groups that are each associated with a layout of the interface, wherein the plurality of groups comprise:
a first group of images associated with a first layout of the interface; and
a second group of images associated with a second layout of the interface;
identifying a first image, of the interface, corresponding to the first layout of the interface; and
comparing, based upon the first group of images, a first portion of the first image with a first portion of a second image of the first group of images to determine user interface testing information associated with the interface of the application.

### Claim 20 (depends on 19)

The device of claim 19, wherein:
the plurality of images comprises screenshots captured via execution of one or more automated tests on the application.
