# US 2024/0004677 A1 · Machine-Learned Models for User Interface Prediction, Generation, and Interaction Understanding

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0004677 A1
- Document ID: US20240004677A1
- Status: Published application
- Kind: A1
- Application number: 18/466,267
- Filed: 2023-09-13
- Published: 2024-01-04
- Earliest claimed: 2021-06-01
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Srinivas Kumar Sunkara; Xiaoxue Zang; Ying Xu; Lijuan Liu; Nevan Holt Wichers; Gabriel Overholt Schubiner; Jindong Chen; Abhinav Kumar Rastogi; Blaise Aguera-Arcas; Zecheng He
- Web page: https://patentgrep.com/patent/US20240004677A1

## CPC

- G06F 9/451: Execution arrangements for user interfaces
- G06F 18/21355: {nonlinear criteria, e.g. embedding a manifold in a Euclidean space}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/24133: {Distances to prototypes}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Generally, the present disclosure is directed to user interface understanding. More particularly, the present disclosure relates to training and utilization of machine-learned models for user interface prediction and/or generation. A machine-learned interface prediction model can be pre-trained using a variety of pre-training tasks for eventual downstream task training and utilization (e.g., interface prediction, interface generation, etc.).

## Claims

### Claim at position 1

-20. (canceled)

### Claim 21

A computer-implemented method for training and utilization of machine-learned models for user interface interaction understanding, comprising:
obtaining, by a computing system comprising one or more computing devices, interface data that comprises a sequence of two or more user interfaces obtained through performance of one or more user interactions which result in generation of the sequence of two or more user interfaces, wherein, for each user interface in the sequence of two or more user interfaces, the interface data comprises one or more interface images depicting the user interface;
determining, by the computing system, a plurality of intermediate embeddings based at least in part on the interface data;
processing, by the computing system, the plurality of intermediate embeddings with a machine-learned interface prediction model to obtain one or more user interface embeddings; and
performing, by the computing system, fine-tuning on the machine-learned interface prediction model based on the one or more user interface embeddings.

### Claim 22 (depends on 21)

The computer-implemented method of claim 21, wherein the interface data further comprises data descriptive of one or more link components, wherein the one or more link components comprise elements of the user interfaces which are the targets of the user interactions.

### Claim 23 (depends on 22)

The computer-implemented method of claim 22, wherein the data descriptive of one or more link components comprises images of the one or more link components.

### Claim 24 (depends on 21)

The computer-implemented method of claim 21, wherein the plurality of intermediate embeddings comprises one or more image embeddings, one or more textual embeddings, and one or more positional embeddings.

### Claim 25 (depends on 21)

The computer-implemented method of claim 21, wherein the plurality of intermediate embeddings comprises one or more vision embeddings.

### Claim 26 (depends on 25)

The computer-implemented method of claim 25, wherein determining the plurality of intermediate embeddings based at least in part on the interface data comprises:
fine-tuning, by the computing system, a neural network to detect user interface components in the one or more interface images; and
determining, by the computing system, the one or more vision embeddings based on the detected user interface components.

### Claim 27 (depends on 26)

The computer-implemented method of claim 26, wherein determining the one or more vision embeddings based on the detected user interface components comprises:
cropping, by the computing system, the one or more user interface images;
providing, by the computing system, the cropped one or more user interface images to a vision encoder; and
receiving, by the computing system, one or more flattened feature maps as the one or more vision embeddings from the vision encoder, the one or more flattened feature maps being generated by the vision encoder based on the one or more cropped user interface images.

### Claim 28 (depends on 21)

The computer-implemented method of claim 21, wherein performing fine-tuning on the machine-learned interface prediction model based on the one or more user interface embeddings comprises:
formatting, by the computing system, a downstream input into one or more segments;
evaluating, by the computing system, a task-specific loss function for the machine-learned interface prediction model; and
adjusting, by the computing system, one or more parameters of the machine-learned interface prediction model based at least in part on the task-specific loss function.

### Claim 29 (depends on 28)

The computer-implemented method of claim 28, wherein the downstream input is a single user interface input, and wherein the interface data for one of the two or more user interfaces is left empty.

### Claim 30 (depends on 28)

The computer-implemented method of claim 28, wherein a specific task associated with the task-specific loss function is natural language input, and wherein the downstream input is a language input comprising a text segment of a first user interface of the two or more user interfaces as a text token and the interface image depicting the first user interface as a corresponding vision token.

### Claim 31 (depends on 21)

The computer-implemented method of claim 21, wherein the interface data further comprises structural data that is indicative of one or more positions of each of a plurality of interface elements included in the user interface, and wherein the structural data for each user interface comprises view hierarchy data associated with the user interface.

### Claim 32

A computing system, comprising:
one or more processors; and
one or more tangible, non-transitory computer readable media storing computer-readable instructions that when executed by one or more processors cause the one or more processors to perform operations, the operations comprising:
obtaining interface data that comprises a sequence of two or more user interfaces obtained through performance of one or more user interactions which result in generation of the sequence of two or more user interfaces, wherein, for each user interface in the sequence of two or more user interfaces, the interface data comprises one or more interface images depicting the user interface;
determining a plurality of intermediate embeddings based at least in part on the interface data;
processing the plurality of intermediate embeddings with a fine-tuned machine-learned interface prediction model to obtain one or more user interface embeddings;
performing a prediction task based at least in part on the one or more user interface embeddings to obtain a prediction output; and
fine-tuning the machine-learned interface prediction model using a task-specific loss function based on the prediction task.

### Claim 33 (depends on 32)

The computing system of claim 32, wherein the prediction task is similar user interface component retrieval, and wherein performing the prediction task comprises:
selecting a candidate user interface component based on a given component associated with at least one user interface embedding of the one or more user-interface embeddings.

### Claim 34 (depends on 33)

The computing system of claim 33, wherein performing the prediction task comprises:
generating a component level embedding for the most similar user interface component using the machine-learned interface prediction model;
determining a dot product of the at least one user interface embedding and the component level embedding; and
determining a similarity score for the candidate user interface component based on the dot product.

### Claim 35 (depends on 32)

The computing system of claim 32, wherein the prediction task is expression component retrieval.

### Claim 36 (depends on 35)

The computing system of claim 35, wherein performing the prediction task comprises:
receiving, as input, a referring expression and an image of a user interface currently displayed; and
selecting, from components of the user interface detected in the image of the user interface, a component referred to by the referring expression as the prediction output.

### Claim 37 (depends on 32)

The computing system of claim 32, wherein the prediction task is icon classification.

### Claim 38 (depends on 37)

The computing system of claim 37, wherein performing the prediction task comprises:
obtaining a first user interface embedding from the one or more user interface embeddings, the first user interface embedding for a user interface component at a first position in at least one user interface of the two or more user interfaces;
using the first user interface embedding as a contextual embedding for the user interface component; and
classifying an icon for the user interface component based on the contextual embedding.

### Claim 39 (depends on 32)

The computing system of claim 32, wherein
the prediction task is application type classification;
the fine-tuned machine-learned interface prediction model has been fine-tuned using one or more user interfaces with ground truth application type labels; and
wherein the prediction output is a type of software application associated with at least one of the two or more user interfaces.

### Claim 40

One or more tangible, non-transitory computer readable media storing computer-readable instructions that when executed by one or more processors cause the one or more processors to perform operations, the operations comprising:
obtaining interface data that comprises a sequence of two or more user interfaces obtained through performance of one or more user interactions which result in generation of the sequence of two or more user interfaces, wherein, for each user interface in the sequence of two or more user interfaces, the interface data comprises one or more interface images depicting the user interface;
determining a plurality of intermediate embeddings based at least in part on the interface data;
processing the plurality of intermediate embeddings with a fine-tuned machine-learned interface prediction model to obtain one or more user interface embeddings;
performing a prediction task based at least in part on the one or more user interface embeddings to obtain a prediction output; and
fine-tuning the machine-learned interface prediction model using a task-specific loss function based on the prediction task.
