Graphical user interface (GUI) noise reduction in a cognitive control framework

An article comprising:

• a non-transitory machine accessible storage medium containing instructions, which when executed, result in reducing graphical user interface (GUI) noise by, during a recording phase of operation of a cognitive control framework system, capturing and storing user input actions and displayed images responsive to the user input actions while the user is using a graphical user interface (GUI) of an application program, and creating a first execution scenario based on the captured user input actions and the displayed images;
• during a first playback phase of operation of the cognitive control framework system, automatically providing the captured user input actions of the first execution scenario to the application program, generating and capturing a first playback image responsive to a particular provided captured user input action, finding an object in the generated first playback image which is to receive a next provided captured user input action, the finding of the object based on image analysis of the generated first playback image using a recorded image of the first execution scenario displayed responsive to the next captured user input action and on at least one search parameter having a broad limit to enable the object to be found in spite of a noise-induced difference between the generated first playback image and the recorded image of the first execution scenario, and recalculating input action data for the provided next captured user input action based on the found object;
• updating the first execution scenario to form a second execution scenario to reduce the noise-induced differences observed in the GUI during the first playback phase, the updates to include the recalculated input action data and the first playback images;
• resetting the at least one search parameter to have a narrow limit compared to the broad limit, the narrow limit to account for the reduction in the noise-induced differences in the GUI;
• and
• playing back the application program according to the second execution scenario, displaying a second playback image in response to a recalculated input action from the second execution scenario, performing image analysis on the second playback image to find the object in the second playback image, the image analysis performed using a first playback image from the second execution scenario which was responsive to the next recalculated input action from the second execution scenario, the finding of the object in the second playback image based on the at least one search parameter having the narrow limit.

A method of automatically controlling execution of an application program having a GUI to reduce GUI noise comprising:

• capturing user input data and images displayed by the GUI responsive to the captured user input data during a recording phase of execution of the application program;
• analyzing the captured user input data and responsive recorded images to generate a first execution scenario during the recording phase;
• generating simulated user input data based on the first execution scenario during a first playback phase of execution of the application program and inputting the simulated user input data to the application program;
• performing image analysis on first playback images displayed by the GUI as a result of processing the simulated user input data during the first playback phase and the recorded images, the image analysis to enable finding, in each first playback image, a next object to receive a next input action according to the first execution scenario, the finding based on a search parameter having a broad limit to enable the next objects to be found in spite of GUI noise conditions that create differences between the recorded images and the first playback images;
• updating the first execution scenario to form a second execution scenario to reduce GUI noise conditions observed during the first playback phase, the updates including the first playback images and recalculated user input data corresponding to the next objects found in the first playback images;
• resetting the search parameter to have a narrow limit as compared to the broad limit to account for the reduced GUI noise;
• and
• playing back the application program according to the second execution scenario and performing image analysis on second playback images displayed by the GUI as a result of processing simulated user input data and first playback images in the second execution scenario to find the next object in each second playback image, the finding of the next object in the second play back images based at least on the reset search parameter having the narrow limit.

An article comprising:

• a non-transitory machine accessible storage medium containing instructions, which when executed, result in automatically controlling execution of an application program having a GUI to reduce GUI noise by, capturing user input data and images displayed by the GUI responsive to the captured user input data during a recording phase of execution of the application program;
• analyzing the captured user input data and responsive recorded images to generate a first execution scenario during the recording phase;
• generating simulated user input data based on the first execution scenario during a first playback phase of execution of the application program and inputting the simulated user input data to the application program;
• performing image analysis on first playback images displayed by the GUI as a result of processing the simulated user input data during the first playback phase and the recorded images, the image analysis to enable finding, in each first playback image, a next object to receive a next input action according to the first execution scenario, the finding based on a search parameter having a broad limit to enable the next objects to be found in spite of GUI noise conditions that create differences between the recorded images and the first playback images;
• updating the first execution scenario to form a second execution scenario to reduce GUI noise conditions observed during the first playback phase, the updates including the first playback images and recalculated user input data corresponding to the next objects found in the first playback images;
• resetting the search parameter to have a narrow limit as compared to the broad limit to account for the reduced GUI noise;
• and
• playing back the application program according to the second execution scenario and performing image analysis on second playback images displayed by the GUI as a result of processing simulated user input data and first sla back ima es in the second execution scenario to find the next object in each second playback image, the finding of the next object in the second play back images based at least on the reset search parameter having the narrow limit.