# US 2018/0024915 A1 · User interface automation framework > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2018/0024915 A1 - Document ID: US20180024915A1 - Status: Published application - Kind: A1 - Application number: 15/214,645 - Filed: 2016-07-20 - Published: 2018-01-25 - Earliest claimed: 2016-07-20 - Assignees: salesforce.com, inc. - Applicants: salesforce.com, inc. - Inventors: Vikas TANEJA - Web page: https://patentgrep.com/patent/US20180024915A1 ## CPC - G06F 11/3688: {for test execution, e.g. scheduling of test suites} - G06F 11/3692: {for test results analysis} - G06F 11/3698: Environments for analysis, debugging or testing of software - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A user interface automation framework is described. A system records multiple user interface screenshots during a session of a user interacting with a user interface application executing on a host computer. The system records metadata associated with the host computer during the session. The system executes a test of the user interface application based on the multiple user interface screenshots and the metadata. ## Claims ### Claim 1 A system for a user interface automation framework, the apparatus comprising: one or more processors; and a non-transitory computer readable medium storing a plurality of instructions, which when executed, cause the one or more processors to: record a plurality of user interface screenshots during a session of a user interacting with a user interface application executing on a host computer; record metadata associated with the host computer during the session; and execute a test of the user interface application based on the plurality of user interface screenshots and the metadata. ### Claim 2 (depends on 1) The system of claim 1, wherein the plurality of user interface screenshots comprise at least one of displayed element location data, selectable element data, tab data, text data, textbox data, and checkbox data. ### Claim 3 (depends on 1) The system of claim 1, wherein the user interacts with the user interface application via a web browser. ### Claim 4 (depends on 1) The system of claim 1, wherein the metadata comprises at least one of a memory utilization, a central processing unit utilization, an input/output throughput, a network bandwidth utilization, and a process execution. ### Claim 5 (depends on 1) The system of claim 1, wherein executing the test comprises comparing metadata associated with a first user interface screenshot of the plurality of user interface screenshots against metadata associated with a second user interface screenshot of the plurality of user interface screenshots. ### Claim 6 (depends on 1) The system of claim 1, wherein executing the test comprises executing a test based on a plurality of user interface screenshots recorded during a subset of the session and metadata recorded during the subset of the session. ### Claim 7 (depends on 1) The system of claim 1, comprising further instructions, which when executed, cause the one or more processors to record any error messages during the session, wherein executing the test further comprises analyzing any error messages. ### Claim 8 A computer program product comprising computer-readable program code to be executed by one or more processors when retrieved from a non-transitory computer-readable medium, the program code including instructions to: record a plurality of user interface screenshots during a session of a user interacting with a user interface application executing on a host computer; record metadata associated with the host computer during the session; and execute a test of the user interface application based on the plurality of user interface screenshots and the metadata. ### Claim 9 (depends on 8) The computer program product of claim 8, wherein the plurality of user interface screenshots comprise at least one of displayed element location data, selectable element data, tab data, text data, textbox data, and checkbox data. ### Claim 10 (depends on 8) The computer program product of claim 8, wherein the user interacts with the user interface application via a web browser. ### Claim 11 (depends on 8) The computer program product of claim 8, wherein the metadata comprises at least one of a memory utilization, a central processing unit utilization, an input/output throughput, a network bandwidth utilization, and a process execution. ### Claim 12 (depends on 8) The computer program product of claim 8, wherein executing the test comprises comparing metadata associated with a first user interface screenshot of the plurality of user interface screenshots against metadata associated with a second user interface screenshot of the plurality of user interface screenshots. ### Claim 13 (depends on 8) The computer program product of claim 8, wherein executing the test comprises executing a test based on a plurality of user interface screenshots recorded during a subset of the session and metadata recorded during the subset of the session. ### Claim 14 (depends on 8) The computer program product of claim 8, wherein the program code comprises further instructions to record any error messages during the session, wherein executing the test further comprises analyzing any error messages. ### Claim 15 A method for a user interface automation framework, the method comprising: recording a plurality of user interface screenshots during a session of a user interacting with a user interface application executing on a host computer; recording metadata associated with the host computer during the session; and executing a test of the user interface application based on the plurality of user interface screenshots and the metadata. ### Claim 16 (depends on 15) The method of claim 15, wherein the plurality of user interface screenshots comprise at least one of displayed element location data, selectable element data, tab data, text data, textbox data, and checkbox data. ### Claim 17 (depends on 15) The method of claim 15, wherein the user interacts with the user interface application via a web browser. ### Claim 18 (depends on 15) The method of claim 15, wherein the metadata comprises at least one of a memory utilization, a central processing unit utilization, an input/output throughput, a network bandwidth utilization, and a process execution. ### Claim 19 (depends on 15) The method of claim 15, wherein executing the test comprises at least one of comparing metadata associated with a first user interface screenshot of the plurality of user interface screenshots against metadata associated with a second user interface screenshot of the plurality of user interface screenshots, and executing a test based on a plurality of user interface screenshots recorded during a subset of the session and metadata recorded during the subset of the session. ### Claim 20 (depends on 15) The method of claim 15, wherein the method further comprises recording any error messages during the session, wherein executing the test further comprises analyzing any error messages.