# US 11,440,201 B2 · Artificial intelligence-based process identification, extraction, and automation for robotic process automation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,440,201 B2 - Document ID: US11440201B2 - Status: Granted - Kind: B2 - Application number: 16/707,564 - Filed: 2019-12-09 - Published: 2021-04-15 - Granted: 2022-09-13 - Earliest claimed: 2019-10-15 - Assignees: UiPath, Inc. - Applicants: UiPath, Inc. - Inventors: Prabhdeep Singh; Christian Berg - Web page: https://patentgrep.com/patent/US11440201B2 ## CPC - B25J 13/006: {by means of a wireless system for controlling one or several manipulators} - G06N 20/00: Machine learning - G06F 8/30: Creation or generation of source code - G06F 9/451: Execution arrangements for user interfaces - G06F 11/3438: {monitoring of user actions (tracking the activity of the user H04L67/535)} - G06F 16/2379: {Updates performed during online database operations; commit processing} - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06F 11/3476: {Data logging (G06F11/14, G06F11/2205 take precedence)} - G06Q 10/0633: Workflow analysis - G06Q 10/10: Office automation; Time management - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR ## Abstract Artificial intelligence (AI)-based process identification, extraction, and automation for robotic process automation (RPA) is disclosed. Listeners may be deployed to user computing systems to collect data pertaining to user actions. The data collected by the listeners may then be sent to one or more servers and be stored in a database. This data may be analyzed by AI layers to recognize patterns of user behavioral processes therein. These recognized processes may then be distilled into respective RPA workflows and deployed to automate the processes. ## Claims ### Claim 1 A system, comprising: a server; and a plurality of user computing systems comprising respective listener applications, the listener applications configured to generate logs comprising user interactions with respective user computing systems and send log data from the logs to the server, wherein the server is configured to: access log data collected from the listeners and run the log data through at least one artificial intelligence (AI) layer, the at least one AI layer configured to process the log data and identify a potential robotic process automation (RPA) process therein, and automatically generate an RPA workflow including the identified RPA process. ### Claim 2 (depends on 1) The system of claim 1, wherein the server is further configured to: generate a robot implementing the RPA workflow; and push the generated robot out to the user computing systems to be executed thereon. ### Claim 3 (depends on 1) The system of claim 1, wherein the at least one AI layer comprises a sequence extraction layer, a clustering detection layer, a visual component detection layer, a text recognition layer, an audio-to-text translation layer, or any combination thereof. ### Claim 4 (depends on 1) The system of claim 1, wherein the server comprises a conductor application and the log data is sent periodically to the conductor application by the listeners of the plurality of user computing systems as part of a heartbeat message. ### Claim 5 (depends on 1) The system of claim 1, wherein the log data is sent to the server once a predetermined amount of log data has been collected, after a predetermined time period has elapsed, or both. ### Claim 6 (depends on 1) The system of claim 1, further comprising: a database communicably coupled to the server, wherein the server is configured to store the received log data from the listeners in the database. ### Claim 7 (depends on 1) The system of claim 1, wherein the server is further configured to: determine similarities between the identified RPA process and a previously existing RPA process executed by a robot on the user computing systems; and replace an RPA workflow of the robot executing the previously existing RPA process with the identified RPA process. ### Claim 8 (depends on 7) The system of claim 7, wherein similarities between the identified RPA process and the previously existing RPA process are determined by a common beginning, a common end, and a predetermined amount of statistical commonality in steps taking in between in the RPA processes. ### Claim 9 (depends on 1) The system of claim 1, wherein the listeners are configured to record where a user clicks on a screen and in which application, keystrokes, which button was clicked, instances of the user switching between applications, focus changes, that an email was sent and what the email pertains to, or any combination thereof. ### Claim 10 (depends on 1) The system of claim 1, wherein the listeners are configured to determine which applications users of the user computing systems are using, what percentage of time the users are using a given application, which features within the applications the users are using, which features within the applications that the users are not using, or any combination thereof. ### Claim 11 A non-transitory computer-readable medium storing a computer program, the computer program configured to cause at least one processor to: access log data collected from respective listener applications of a plurality of user computing systems; run the log data through at least one artificial intelligence (AI) layer, the at least one AI layer configured to process the log data and identify a potential robotic process automation (RPA) process therein, and automatically generate an RPA workflow including the identified RPA process. ### Claim 12 (depends on 11) The non-transitory computer-readable medium of claim 11, wherein the computer program is further configured to cause the at least one processor to: generate a robot implementing the RPA workflow; and push the generated robot out to the user computing systems to be executed thereon. ### Claim 13 (depends on 11) The non-transitory computer-readable medium of claim 11, wherein the at least one AI layer comprises a sequence extraction layer, a clustering detection layer, a visual component detection layer, a text recognition layer, an audio-to-text translation layer, or any combination thereof. ### Claim 14 (depends on 11) The non-transitory computer-readable medium of claim 11, wherein the computer program is further configured to cause the at least one processor to: determine similarities between the identified RPA process and a previously existing RPA process executed by a robot on the user computing systems; and replace an RPA workflow of the robot executing the previously existing RPA process with the identified RPA process. ### Claim 15 (depends on 11) The non-transitory computer-readable medium of claim 11, wherein similarities between the identified RPA process and the previously existing RPA process are determined by a common beginning, a common end, and a predetermined amount of statistical commonality in steps taking in between in the RPA processes. ### Claim 16 A computer-implemented method, comprising: generating, by a listener, a log comprising user interactions with a user computing system; and sending log data from the log to a server, by the listener, wherein the log data comprises where a user clicks on a screen and in which application, keystrokes, which button was clicked, instances of the user switching between applications, focus changes, that an email was sent and what the email pertains to, or any combination thereof. ### Claim 17 (depends on 16) The computer-implemented method of claim 16, wherein the log data is sent periodically to a conductor application of the server as part of a heartbeat message. ### Claim 18 (depends on 16) The computer-implemented method of claim 16, wherein the log data is sent to the server once a predetermined amount of log data has been collected, after a predetermined time period has elapsed, or both. ### Claim 19 (depends on 16) The computer-implemented method of claim 16, wherein the user interactions comprise which applications the user is using, what percentage of time the user is using a given application, which features within the applications the user is using, which features within the applications that the user is not using, or any combination thereof. ### Claim 20 (depends on 16) The computer-implemented method of claim 16, wherein the user interactions comprise which features within the applications the user is using and which features within the applications that the user is not using.