# US 2021/0342247 A1 · Mathematical models of graphical user interfaces

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342247 A1
- Document ID: US20210342247A1
- Status: Published application
- Kind: A1
- Application number: 16/866,179
- Filed: 2020-05-04
- Published: 2021-11-04
- Earliest claimed: 2020-05-04
- Assignees: Asapp, Inc.
- Applicants: Asapp, Inc.
- Inventors: Daniel Alfredo Ciolek; Clemens Georg Benedict Rosenbaum; Adrian Philip Botta
- Web page: https://patentgrep.com/patent/US20210342247A1

## CPC

- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/0483: Interaction with page-structured environments, e.g. book metaphor
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 11/302: {where the computing system component is a software system}
- G06F 11/3409: {for performance assessment}
- G06F 11/3438: {monitoring of user actions (tracking the activity of the user H04L67/535)}
- G06F 11/3447: {Performance evaluation by modeling}
- G06F 11/3452: {Performance evaluation by statistical analysis}
- G06F 11/3466: {Performance evaluation by tracing or monitoring}
- G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)}
- G06F 18/29: {Graphical models, e.g. Bayesian networks}
- G06F 40/14: Tree-structured documents (parsing G06F40/205; validation G06F40/226)
- G06F 40/166: Editing, e.g. inserting or deleting
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06F 2201/81: Threshold
- G06F 2201/865: Monitoring of software
- G06F 2201/88: Monitoring involving counting
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A graph model of a graphical user interface (GUI) may be generated by processing usage data of the GUI where the usage data comprises sequences of GUI pages and actions between GUI pages. The nodes of the graph model may be determined by obtaining GUI pages from the usage data, identifying dynamic GUI elements in the GUI pages, generating canonical GUI pages by modifying the GUI pages using the dynamic GUI elements, and creating graph nodes using the canonical GUI pages. The edges of the graph may be determined by processing actions from the GUI data that were performed by users to transition from one GUI page to another GUI page. The graph model of the GUI may be used for any appropriate application, such as determining statistics relating to the GUI or statistics relating to individual users of the GUI.

## Claims

### Claim 1

A computer-implemented method, comprising:
obtaining usage data of a graphical user interface (GUI) comprising sequences of GUI pages and actions between GUI pages;
generating a graph model of the GUI, wherein generating the graph model comprises:
obtaining a plurality of GUI pages from the usage data,
identifying a plurality of dynamic GUI elements from the plurality of GUI pages,
generating a plurality of canonical GUI pages by modifying the plurality of GUI pages using the plurality of dynamic GUI elements,
generating a plurality of graph nodes using the plurality of canonical GUI pages,
obtaining a plurality of actions from the usage data, and
generating a plurality of graph edges using the plurality of actions; and
processing the usage data with the graph model to determine a statistic for a portion of the graph model.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the GUI corresponds to a website and the GUI pages are HTML pages.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, comprising:
generating a predictive model of the GUI; and
processing a GUI page with the predictive model to select a graph node from the plurality of graph nodes.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein the predictive model comprises a neural network.

### Claim 5 (depends on 3)

The computer-implemented method of claim 3, wherein the predictive model is trained by processing the usage data of the GUI.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the portion of the graph model comprises an edge, a node, or a subgraph.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein determining the statistic comprises determining a number of times that the usage data includes:
an action corresponding to a first edge of the graph model; or
a GUI page corresponding to a first node of the graph model.

### Claim 8

A system, comprising:
at least one server computer comprising at least one processor and at least one memory, the at least one server computer configured to:
obtain usage data of a graphical user interface (GUI) comprising sequences of GUI pages and actions between GUI pages;
generate a graph model of the GUI, wherein generating the graph model comprises:
obtaining a plurality of GUI pages from the usage data,
identifying a plurality of dynamic GUI elements from the plurality of GUI pages,
generating a plurality of canonical GUI pages by modifying the plurality of GUI pages using the plurality of dynamic GUI elements,
generating a plurality of graph nodes using the plurality of canonical GUI pages,
obtaining a plurality of actions from the usage data, and
generating a plurality of graph edges using the plurality of actions; and
process the usage data with the graph model to determine a statistic for a portion of the graph model.

### Claim 9 (depends on 8)

The system of claim 8, wherein the statistic corresponds to an amount of time between arriving at a GUI page corresponding to a first node and a user performing an action corresponding to a first edge.

### Claim 10 (depends on 8)

The system of claim 8, wherein the statistic corresponds to an amount of time between arriving at a GUI page corresponding to a first node and a user performing an action to leave the GUI page corresponding to the first node.

### Claim 11 (depends on 10)

The system of claim 10, wherein the statistic is a total amount of time, a median amount of time, or an average amount of time.

### Claim 12 (depends on 8)

The system of claim 8, wherein identifying the plurality of dynamic GUI elements comprises computing a conditional probability of a GUI element given its parent GUI element.

### Claim 13 (depends on 8)

The system of claim 8, wherein generating the plurality of canonical GUI pages comprises modifying or removing the plurality of dynamic GUI elements.

### Claim 14 (depends on 8)

The system of claim 8, generating the plurality of canonical GUI pages comprises replacing a dynamic GUI element or a portion of a dynamic GUI element with a wildcard indicating dynamic information.

### Claim 15

One or more non-transitory, computer-readable media comprising computer-executable instructions that, when executed, cause at least one processor to perform actions comprising:
obtaining usage data of a graphical user interface (GUI) comprising sequences of GUI pages and actions between GUI pages;
generating a graph model of the GUI, wherein generating the graph model comprises:
obtaining a plurality of GUI pages from the usage data,
identifying a plurality of dynamic GUI elements from the plurality of GUI pages,
generating a plurality of canonical GUI pages by modifying the plurality of GUI pages using the plurality of dynamic GUI elements,
generating a plurality of graph nodes using the plurality of canonical GUI pages,
obtaining a plurality of actions from the usage data, and
generating a plurality of graph edges using the plurality of actions; and
processing the usage data with the graph model to determine a statistic for a portion of the graph model.

### Claim 16 (depends on 15)

The one or more non-transitory, computer-readable media of claim 15, wherein processing the usage data with the graph model to determine the statistic comprises processing usage data corresponding to a first user of a plurality of users.

### Claim 17 (depends on 16)

The one or more non-transitory, computer-readable media of claim 16, wherein the plurality of users comprise customer service agents.

### Claim 18 (depends on 15)

The one or more non-transitory, computer-readable media of claim 15, wherein a graph edge is associated with a probability.

### Claim 19 (depends on 15)

The one or more non-transitory, computer-readable media of claim 15, the actions comprising:
generating a predictive model of the GUI; and
processing a GUI page with the predictive model to select a graph node from the plurality of graph nodes.

### Claim 20 (depends on 15)

The one or more non-transitory, computer-readable media of claim 15, wherein identifying the plurality of dynamic GUI elements comprises computing a conditional probability of a GUI element given its parent GUI element.
