# US 12,182,557 B2 · Methods and systems for automating computer application tasks using application guides, markups and computer vision > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,557 B2 - Document ID: US12182557B2 - Status: Granted - Kind: B2 - Application number: 16/672,499 - Filed: 2019-11-03 - Published: 2020-07-16 - Granted: 2024-12-31 - Earliest claimed: 2018-11-02 - Assignees: EdCast Inc. - Applicants: EdCast Inc. - Inventors: Nilesh Pachpande; Kaushal Mehta - Web page: https://patentgrep.com/patent/US12182557B2 ## CPC - G06F 8/61: Installation - G06F 40/143: Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD] - G06T 1/0007: {Image acquisition} - G06F 9/453: {Help systems} - G06F 11/07: Responding to the occurrence of a fault, e.g. fault tolerance - G06F 11/079: {Root cause analysis, i.e. error or fault diagnosis (in a hardware test environment G06F11/22; in a software test environment G06F11/36)} - G06F 11/36: Prevention of errors by analysis, debugging or testing of software - G06F 11/3688: {for test execution, e.g. scheduling of test suites} - G06F 40/14: Tree-structured documents (parsing G06F40/205; validation G06F40/226) - G06Q 30/0625: {by formulating product or service queries, e.g. using keywords or predefined options} - G10L 15/16: using artificial neural networks - H04L 67/02: based on web technology, e.g. hypertext transfer protocol [HTTP] - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract In one aspect, a computerized method useful for automating computer application tasks using application guides, markups and computer vision includes the step of automating a human-implemented computer task. This includes determining a human workflow via a set of images. This also includes automatically deriving the workflow based on the set of images and a set of human user inputs. The method includes the step of, based on the automated human-implemented computer task, automatically implementing a decisions on behalf of a human user. The method includes the step of, implementing the automated human-implemented computer task using an in-application guide or a computer-vision step to determine a task the human user has performed and then performing the same task as the human user. ## Claims ### Claim 1 A computerized method useful for automating computer application tasks using application guides, markups and computer vision comprising: automating a human-implemented computer task by: installing an application software on a web browser via at least one web-browser extension, wherein the application software records a software workflow when instructed by the human user based on a start action and an end action and records one or more API responses from an operating system; determining a human workflow via a set of images; deriving a signal to teach a computer system by reviewing images from a specified number of transactions wherein a triggering event has happened; and automatically deriving the human workflow based on the set of images and a set of human user inputs and an Application Program Interface (API) response from the operating system, wherein the API response identifies one or more of a current window, a current application in focus, and a current screen in focus; based on the automated human-implemented computer task, automatically implementing a decision on behalf of a human user; and implementing the automated human-implemented computer task using an in-application guide or a computer-vision step to determine a task the human user has performed and then performing the same task as the human user, wherein the transaction is one or more of expense reporting or performing a business task based on instructions and decision criteria. ### Claim 2 (depends on 1) The computerized method of claim 1, wherein the computer-vision step comprises using a computer vision system to understand a set of text behind a set of buttons and gather further meta data about the current application in focus, wherein the set of text comprises a Hypertext Markup Language (HTML) element of the set of buttons and an API response of the set of buttons. ### Claim 3 (depends on 2) The computerized method of claim 2, further comprising: wherein a mouse click results in a capture of an underlying screen and a specified background information, wherein a record of every screen where the mouse click is detected is used to capture a set of screenshots to create the software workflow. ### Claim 4 (depends on 3) The computerized method of claim 3, wherein a computer automatically implements the decision on behalf of the human user to perform a same task as the human user and take a decision based on human user-provided intelligence at a specified task creation time. ### Claim 5 (depends on 3) The computerized method of claim 3, further comprising installing an application software on a mobile machine via a mobile application. ### Claim 6 (depends on 5) The computerized method of claim 5, wherein the application software records a software workflow of the current window, the current screen in focus and the current application in focus when instructed by the human user based on a start action and end action or a record of the API response. ### Claim 7 (depends on 5) The computerized method of claim 5, wherein the specified background information comprises a Document Object Model of a webpage accessed by the web browser, and wherein the specified background information comprises an associated HTML element in the web browser. ### Claim 8 (depends on 1) The computerized method of claim 1, wherein the human-implemented computer task is started in an application installed on a first computer, wherein the task ends on a second computer. ### Claim 9 A computer system useful for automating computer application tasks using application guides, markups and computer vision comprising: a processor; an operating system, the operating system executing on the processor; a memory containing instructions when executed on the processor, causes the processor to perform operations that: automate a human-implemented computer task by: installing an application software on a web browser via at least one web-browser extension; recording a software workflow by recording every screen wherein a triggering event is detected when instructed to do based on start and end actions of the workflow, including recording one or more Application Program Interface (API) responses from the operating system; determining a human workflow via a set of images; deriving a signal to teach a computer system by reviewing images from a specified number of transactions wherein a triggering event has happened; and automatically deriving the human workflow based on the set of images and a set of human user inputs and the API response from the operating system, wherein the API response identifies one or more of a current window, a current application in focus, and a current screen in focus; based on the automated human-implemented computer task, automatically implementing a decision on behalf of a human user; and implementing the automated human-implemented computer task using an in-application guide or a computer-vision step to determine a task the human user has performed and then performing the same task as the human user, wherein the transaction is one or more of expense reporting or performing a business task based on instructions and decision criteria. ### Claim 10 (depends on 9) The computer system of claim 9, wherein the computer-vision step comprising comprises using a computer vision system to understand a set of text behind a set of buttons and gather further meta data about the current application in focus, wherein the set of text comprises a Hypertext Markup Language (HTML) element of the set of buttons and an API response of the set of buttons, wherein the transaction is expense reporting. ### Claim 11 (depends on 10) The computer system of claim 10, wherein a mouse click results in a capture of an underlying screen and a specified background information, wherein a record of every screen where the mouse click is detected is used to capture a set of screenshots to create the software workflow; wherein a computer automatically implement the decision on behalf of the human user to perform a same task as the human user and take a decision based on human user-provided intelligence at a specified task creation time, installing the application software on a mobile machine via a mobile application, installing the application software on a computer machine via a desktop application. ### Claim 12 (depends on 11) The computer system of claim 11, wherein the application software records a software workflow of the current window, the current screen in focus and the current application in focus when instructed by the human user based on the start action and the end action or the API response. ### Claim 13 (depends on 12) The computer system of claim 12, wherein the specified background information comprises a Document Object Model of a webpage accessed by the web browser, and wherein the specified background information comprises an associated HTML element in the web browser. ### Claim 14 (depends on 9) The computer system of claim 9, wherein the human-implemented computer task is started in an application installed on a first computer, wherein the task ends on a second computer.