Methods and systems for automating computer application tasks using application guides, markups and computer vision

A computer system useful for automating computer application tasks using application guides, markups and computer vision comprising:

• a processor;
• an operating system, the operating system executing on the processor;
• a memory containing instructions when executed on the processor, causes the processor to perform operations that:
• automate a human-implemented computer task by:
• installing an application software on a web browser via at least one web-browser extension;
• recording a software workflow by recording every screen wherein a triggering event is detected when instructed to do based on start and end actions of the workflow, including recording one or more Application Program Interface (API) responses from the operating system;
• determining a human workflow via a set of images;
• deriving a signal to teach a computer system by reviewing images from a specified number of transactions wherein a triggering event has happened;
• and
• automatically deriving the human workflow based on the set of images and a set of human user inputs and the API response from the operating system, wherein the API response identifies one or more of a current window, a current application in focus, and a current screen in focus;
• based on the automated human-implemented computer task, automatically implementing a decision on behalf of a human user;
• and
• implementing the automated human-implemented computer task using an in-application guide or a computer-vision step to determine a task the human user has performed and then performing the same task as the human user, wherein the transaction is one or more of expense reporting or performing a business task based on instructions and decision criteria.