# US 2019/0377669 A1 · Framework for visual audit emulation for application

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0377669 A1
- Document ID: US20190377669A1
- Status: Published application
- Kind: A1
- Application number: 16/434,625
- Filed: 2019-06-07
- Published: 2019-12-12
- Earliest claimed: 2018-06-07
- Assignees: UST Global (Singapore) Pte. Ltd.
- Applicants: UST Global (Singapore) Pte. Ltd.
- Inventors: Aby Jacob; Satheesh Gopalakrishna Pillai
- Web page: https://patentgrep.com/patent/US20190377669A1

## CPC

- G06F 11/3438: {monitoring of user actions (tracking the activity of the user H04L67/535)}
- G06F 9/451: Execution arrangements for user interfaces
- G06F 11/3079: {the data filtering being achieved by reporting only the changes of the monitored data}
- G06F 11/3082: {the data filtering being achieved by aggregating or compressing the monitored data}
- G06F 21/50: Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F 11/3072: {where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting}
- G06F 11/3476: {Data logging (G06F11/14, G06F11/2205 take precedence)}
- G06F 2201/835: Timestamp
- G06F 2201/86: Event-based monitoring
- G09B 7/00: Electrically-operated teaching apparatus or devices working with questions and answers (mechanically operated G09B3/00; computing arrangements G06F)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)

## Abstract

A system and method to generate an audit trail based on operation of a target application. The system includes a computing device operable to execute the target application. The target software generates user interfaces audit data in response to user inputs. The audit data generated by the target application is stored. An audit visualization framework reads the audit database and creates a video playback file of user actions that occur as the user interacts with the audited target application.

## Claims

### Claim 1

A system to generate an audit trail based on operation of a target application, the system comprising:
a computing device operable to execute the target application, the computing device including an input device and a display, wherein the target application generates a plurality of user interfaces on the display and audit data in response to user inputs received from the input device;
a memory coupled to the computing device to store the audit data generated by the target application; and
an auditing device coupled to the memory and the computing device, the auditing device operable to generate a video file of the operation of the target application based on one of a plurality of templates, each of the templates corresponding to one of the user interfaces, and the audit data generated from a user input received from the input device, the user input associated with one of the user interfaces.

### Claim 2 (depends on 1)

The system of claim 1, further comprising a display coupled to the auditing device for playing the generated video file of the operation of the target application.

### Claim 3 (depends on 1)

The system of claim 1, further comprising a database with reference data generated by the computing device in the execution of the target application, wherein the reference data is incorporated in the generated video file.

### Claim 4 (depends on 1)

The system of claim 1, further comprising a network coupled to the computing device, memory and auditing device.

### Claim 5 (depends on 1)

The system of claim 1, wherein the audit data is extracted from the target application and converted for storage in the memory by one of JDBC, ODBC, Web services or API.

### Claim 6 (depends on 1)

The system of claim 1, wherein the audit data includes a timestamp for individual actions.

### Claim 7 (depends on 1)

The system of claim 1, wherein input device is a mouse, and wherein the video file includes inputs that track mouse movement on the one of the user interfaces.

### Claim 8 (depends on 1)

The system of claim 1, wherein the video file includes frames that each include one of the templates showing an interface of the target application and a dynamic element derived from the audit data.

### Claim 9 (depends on 1)

The system of claim 1, further comprising an audit visualization framework that dissects each of the user interfaces into static and dynamic elements, wherein the static elements are those that remain the same irrespective of a user of the target application, a time of login, or a state of the user or of the target application information, and wherein the dynamic elements are variable elements in the user interface, the audit visualization framework being configured to recreate the dynamic elements and overlay content of the dynamic elements to recreate a state or appearance of any of the user interfaces and interaction by the user with any of the user interfaces at any given instant of time in the past.

### Claim 10

A method of providing a visual audit trail of operating a target application, the method comprising:
executing the target application from a computing device, the target application generating audit data associated with the actions of a user operating the target application;
storing the audit data in a database on a storage device;
accessing a template associated with a user interface generated by the target application; and
generating an audit video file from the template and the audit data, the audit video indicative of the actions of the user operating the target application.

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
dissecting the user interface into static and dynamic elements, wherein the static elements are those that remain the same irrespective of a user of the target application, a time of login, or a state of the user or of the target application information, and wherein the dynamic elements are variable elements in the user interface; and
recreating the dynamic elements and overlay content of the dynamic elements to recreate a state or appearance of the user interface and interaction by the user with the user interface at any given instant of time in the past.

### Claim 12 (depends on 10)

The method of claim 10, further comprising playing the generated video file of the operation of the target application on a display.

### Claim 13 (depends on 10)

The method of claim 10, wherein the audit data is extracted from the target application and converted for storage by one of JDBC, ODBC, Web services or API.

### Claim 14 (depends on 10)

The method of claim 10, wherein the audit data includes a timestamp for individual actions.

### Claim 15 (depends on 10)

The method of claim 10, wherein the actions of the user include operation of an input device.

### Claim 16 (depends on 15)

The method of claim 15, wherein the input device is a mouse, and wherein the video file includes inputs that track mouse movement on the one of the user interfaces.

### Claim 17 (depends on 10)

The method of claim 10, wherein the audit video file includes frames that each include one of the templates showing an interface of the target and a dynamic element derived from the audit data.
