Recording/playback tools for UI-based applications
AssigneeMicrosoft Corporation
InventorDmitri A. Klementiev
Life of this patent
Estimating the term…
Abstract
Techniques and tools are described for recording input from user actions in a user interface (UI) and replicating the UI activity on a computing device. When recording and replicating UI activity, these techniques and tools improve the readability of the recorded input data and the reliability of playback. The techniques and tools may be used in combination or separately. For example, a recording tool uses a set of filters, which aggregates recorded data into basic, readable primitive methods. The recording tool converts the aggregated data into playback code by converting the playback primitive methods into corresponding computer language instructions. A playback tool may then replicate the initial recorded UI activity by playing back the computer language instructions.
Claims28
A computer-readable storage medium storing computer-executable instructions for causing a computer system programmed thereby to perform a method of selectively recording meaningful user activity in a graphical user interface, the method comprising:
- receiving a plurality of actions input for a graphical user interface;
- aggregating the plurality of actions using a set of one or more filters into aggregated actions, wherein at least one filter is an absorbing filter, the absorbing filter including a first user-defined rule and a second user-defined rule, the first rule including a reference to the second rule, to define user interface data actions defined by the first rule that are replaced by later user interface data actions defined by the second rule;
- and
- converting the aggregated actions into a set of user interface recording/playback primitives, such that playing back the user interface recording/playback primitives replicates the meaningful user activity recorded in the graphical user interface,
- wherein at least one of the set of user interface recording/playback primitives is a verification method which determines during playback whether a user interface element is available and in a correct state for playback of a next one of the set of user interface recording/playback primitives.
A computer-readable storage medium storing computer-executable instructions for causing a computer system programmed thereby to perform a method of generating reliable and readable playback code based on user activity in a graphical user interface, the method comprising:
- receiving and recording a plurality of actions input for a graphical user interface from one or more input devices;
- receiving and recording user element value change events for user elements in the graphical user interface from operating system events on the computer system;
- classifying the plurality of recorded actions and user element value change events into mouse actions, keyboard actions, and other actions, wherein other actions comprise select events, value change events, and expand events, wherein mouse actions are actions received from a mouse, keyboard actions are actions received from a keyboard, select events are actions received when a user selects or unselects a user interface element, and expand events are actions received when an expandable user interface element is expanded or collapsed;
- converting one or more of the classified actions and user element value change events into a set of playback primitives;
- wherein at least two of the plurality of actions are represented as a single playback primitive, at least one playback primitive includes a user element value change event, and wherein only a most recent expand event in a series of expand events is converted to a playback primitive;
- and
- generating playback code based on the set of playback primitives.
A tool for recording user interface activity, the tool comprising:
- one or more processing units;
- one or more memory units;
- one or more input devices;
- one or more output devices;
- and
- storage for storing computer-executable instructions for recording user interface activity using:
- a module for collecting data associated with user interface activity;
- a module for screening out certain categories of the data collected by the module for collecting user interface element data to produce filtered user interface element data and for simplifying the filtered user interface element data to produce filtered, simplified data, the module collecting a user interface element identifier, an identifier of the parent, and a container identifier for each user interface element represented;
- a module for converting the filtered, simplified data into a set of user interface playback primitives;
- and
- a module for converting the user interface playback primitives into output code and outputting the output code, the output code the being usable in a module for replicating at least a portion of the user interface activity;
- wherein the module for collecting data associated with user interface activity further comprises a tool within the module for collecting only the last user interface element expand event in a series of user interface element expand events, wherein the module places the series of user interface element expand events on a stack, and wherein collecting a non-expand event causes the module to identify the last user interface element expand event using the stack.
A computer-readable storage medium storing computer-executable instructions for causing a computer system programmed thereby to perform a method of reliably replicating user activity in a graphical user interface, the method comprising:
- receiving playback code based on a set of playback primitives, the playback code representing a series of user actions previously input into a first graphical user interface;
- and
- processing the playback code to play back the series of user actions through a second graphic user interface comprising:
- retrieving a next user action in the series of user actions,
- if a container is associated with the next user action, waiting for the container to open,
- if a playback primitive in the playback code includes an indication that a user interface element associated with the next user action should be determined to be visible before proceeding, waiting for the user interface element associated with the next user action to be visible, and
- if the user interface element has an associated window, then waiting for the associated window to be visible, enabled, and ready for input;
- if a failure occurs during waiting for the container to open, waiting for the user interface element, or waiting for the associated window, then waiting for a predetermined time to avoid synchronization problems;
- and
- performing the next user action.
Dependencies are inferred from the claim text.