Extracting features from screen images for task mining

A system comprising:

• a memory storing computer instructions;
• and
• at least one processor configured to execute the computer instructions, the computer instructions configured to cause the at least one processor to perform operations of:
• receiving a screen image depicting a user interface of a computing system;
• preprocessing the screen image to generate a preprocessed screen image and processing results;
• extracting image features from the preprocessed screen image using a first machine learning based network;
• extracting text features and control element features from the processing results using a second machine learning based network;
• encoding the text features and the control element features using a third machine learning based network to generate representative features of the screen image;
• and
• performing a task on the screen image based on one or more of the image features, the text features, the control element features, or the representative features.

A non-transitory computer-readable medium storing computer program instructions, the computer program instructions, when execution on at least one processor, cause the at least one processor to perform operations comprising:

• receiving a screen image depicting a user interface of a computing system;
• preprocessing the screen image to generate a preprocessed screen image and processing results;
• extracting image features from the preprocessed screen image using a first machine learning based network;
• extracting text features and control element features from the processing results using a second machine learning based network;
• encoding the text features and the control element features using a third machine learning based network to generate representative features of the screen image;
• and
• performing a task on the screen image based on one or more of the image features, the text features, the control element features, or the representative features.