# US 12,118,436 B2 · Automated user application tracking and validation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,436 B2
- Document ID: US12118436B2
- Status: Granted
- Kind: B2
- Application number: 17/156,069
- Filed: 2021-01-22
- Published: 2022-07-28
- Granted: 2024-10-15
- Earliest claimed: 2021-01-22
- Assignees: Disney Enterprises, Inc.
- Applicants: Disney Enterprises, Inc.
- Inventors: Dustin Stefan Hamerla; Abrasham Chowdhury; Christopher Adam Boyle; Joseph H. I. Bird; Melissa Ashley Moyer; Kyle Patrick Baker; Taylor C. Wells; Vaibhav Jajoo
- Web page: https://patentgrep.com/patent/US12118436B2

## CPC

- G06N 20/00: Machine learning
- G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F 11/3438: {monitoring of user actions (tracking the activity of the user H04L67/535)}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/0485: Scrolling or panning
- G06F 2201/86: Event-based monitoring
- G06F 2218/00: Aspects of pattern recognition specially adapted for signal processing
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A system includes a computing platform including a hardware processor and a system memory storing a software code. The hardware processor is configured to execute the software code to track interactions with a user application during use of the user application, generate, based on tracking the interactions, interaction data identifying multiple interaction events during the use, and perform a validity assessment of the interaction data. The hardware processor is further configured to execute the software code to identify, based on the validity assessment, one or more anomalies in the interaction data, and output, based on identifying the one or more anomalies in the interaction data, one or more of the interaction events corresponding respectively to the one or more anomalies.

## Claims

### Claim 1

A system comprising:
a computing platform including a hardware processor and a system memory storing a software code;
the hardware processor configured to execute the software code to:
track interactions with a user application during a use of the user application;
generate, based on tracking the interactions, interaction data identifying a plurality of interaction events during the use, the interaction data including a plurality of states of the user application each corresponding to one of the plurality of interaction events, wherein each of the plurality of states of the user application is assigned at least one key that is included in the interaction data to link the plurality of states to recreate the interactions of the user;
perform a validity assessment of the interaction data;
identify, based on the validity assessment, one or more anomalies in the interaction data; and
output, based on identifying the one or more anomalies in the interaction data, one or more of the plurality of interaction events corresponding respectively to the one or more anomalies.

### Claim 2 (depends on 1)

The system of claim 1, wherein the hardware processor is configured to perform the validity assessment in response to the use of the user application by a user emulation bot, wherein the plurality of interactions are scripted interactions executed by the user emulation bot.

### Claim 3 (depends on 1)

The system of claim 1, wherein the user application is configured to provide a graphical user interface (GUI), and wherein the plurality of states include a display state of the GUI.

### Claim 4 (depends on 3)

The system of claim 3, wherein the at least one key includes a primary key and a foreign key assigned to the display state of the GUI.

### Claim 5 (depends on 1)

The system of claim 1, wherein the plurality of interaction events include at least one of a page view, a click-through, a scrolling input, a text entry, a menu selection, or a return-to-previous-page input.

### Claim 6 (depends on 1)

The system of claim 1, wherein the hardware processor is further configured to execute the software code to perform the validity assessment using a sequence of the plurality of interaction events, by:
generating an interaction vector corresponding to the sequence; and
determining a validity score for the interaction vector.

### Claim 7 (depends on 6)

The system of claim 6, wherein the hardware processor is further configured to execute the software code to perform the validity assessment by:
identifying an expected-value vector using a first interaction event of the sequence; and
comparing the interaction vector to the expected-value vector to determine the validity score for the interaction vector.

### Claim 8 (depends on 6)

The system of claim 6, wherein the validity score is determined by applying the interaction vector to an anomaly prediction model.

### Claim 9 (depends on 6)

The system of claim 6, wherein, for each interaction event of the plurality of interaction events of the sequence, the interaction vector encodes a first instance of the each interaction event in the sequence and a cumulative sum of instances of the each interaction event in the sequence.

### Claim 10 (depends on 1)

The system of claim 1, wherein the hardware processor is further configured to execute the software code to:
train an anomaly prediction machine learning model, using the one or more of the plurality of interaction events corresponding respectively to the one or more anomalies;
wherein the validity assessment of the interaction data is performed using the anomaly prediction machine learning model.

### Claim 11

A method for use by a system including a computing platform having a hardware processor and a system memory storing a software code, the method comprising:
tracking, by the software code executed by the hardware processor, interactions with a user application during a use of the user application;
generating, by the software code executed by the hardware processor and based on tracking the interactions, interaction data identifying a plurality of interaction events during the use, the interaction data including a plurality of states of the user application each corresponding to one of the plurality of interaction events, wherein each of the plurality of states of the user application is assigned at least one key that is included in the interaction data to link the plurality of states to recreate the interactions of the user;
performing, by the software code executed by the hardware processor and using the validation subsystem, a validity assessment of the interaction data;
identifying, by the software code executed by the hardware processor and based on the validity assessment, one or more anomalies in the interaction data; and
outputting, by the software code executed by the hardware processor and based on identifying the one or more anomalies in the interaction data, one or more of the interaction events corresponding respectively to the one or more anomalies.

### Claim 12 (depends on 11)

The method of claim 11, wherein the hardware processor is configured to perform the validity assessment in response to the use of the user application by a user emulation bot, wherein the plurality of interactions are scripted interactions executed by the user emulation bot.

### Claim 13 (depends on 11)

The method of claim 11, wherein the user application is configured to provide a graphical user interface (GUI), and wherein the plurality of states include a display state of the GUI.

### Claim 14 (depends on 13)

The method of claim 13, wherein the at least one key includes a primary key and a foreign key assigned to the display state of the GUI.

### Claim 15 (depends on 11)

The method of claim 11, wherein the plurality of interaction events include at least one of a page view, a click-through, a scrolling input, a text entry, a menu selection, or a return-to-previous-page input.

### Claim 16 (depends on 11)

The method of claim 11, wherein the validity assessment is performed using a sequence of the plurality of interaction events, by:
generating, by the software code executed by the hardware processor, an interaction vector corresponding to the sequence; and
determining, by the software code executed by the hardware processor, a validity score for the interaction vector.

### Claim 17 (depends on 16)

The method of claim 16, wherein the validity assessment is further performed by:
identifying, by the software code executed by the hardware processor, an expected-value vector using a first interaction event of the sequence; and
comparing, by the software code executed by the hardware processor, the interaction vector to the expected-value vector to determine the validity score for the interaction vector.

### Claim 18 (depends on 16)

The method of claim 16, wherein the validity score is determined by applying the interaction vector to an anomaly prediction model.

### Claim 19 (depends on 16)

The method of claim 16, wherein, for each interaction event of the plurality of interaction events of the sequence, the interaction vector encodes a first instance of the each interaction event in the sequence and a cumulative sum of instances of the each interaction event in the sequence.

### Claim 20 (depends on 11)

The method of claim 11, further comprising:
training an anomaly prediction machine learning model, using the one or more of the plurality of interaction events corresponding respectively to the one or more anomalies;
wherein performing the validity assessment of the interaction data uses the anomaly prediction machine learning model.