Method for robotic process automation, apparatus and system for executing the method

A robotic processing automation apparatus comprising:

• a processor,
• wherein the processor is configured to perform operations of:
• transmitting to a remote apparatus, a screen stream request for requesting a screen of the remote apparatus, the screen of the remote apparatus including a screen of a target program installed in the remote apparatus, receiving from the remote apparatus, a screen stream corresponding to the screen stream request, and displaying a screen of the robotic processing automation apparatus including a remote terminal screen, the remote terminal screen including the received screen stream;
• and
• transmitting to the remote apparatus, when a user event occurs in the screen stream displayed in the screen of the robotic processing automation apparatus, an information inquiry request for an object related to the user event, receiving from the remote apparatus, object-related information corresponding to the information inquiry request, and generating an automation script for automatically executing the target program based on the screen stream and the object-related information, the information inquiry request including coordinate information of the object on the screen of the remote apparatus, the coordinate information of the object being obtained based on coordinates of the object on the screen of the robotic processing automation apparatus,
• wherein the operation of obtaining the coordinate information of the object comprises:
• extracting first coordinates of the object from the screen of the robotic processing automation apparatus;
• converting the first coordinates of the object into second coordinates on the remote terminal screen based on a relative position between the screen of the robotic processing automation apparatus and the remote terminal screen;
• and
• converting the second coordinates into third coordinates on the screen of the remote apparatus using a resolution of the remote terminal screen and a resolution of the screen of the remote apparatus.

A robotic processing automation system comprising:

• a robotic processing automation apparatus configured to:
• transmit a screen stream request for requesting a screen of a remote apparatus, the screen of the remote apparatus including an execution screen of a target program installed in the remote apparatus,
• receive a screen stream corresponding to the screen stream request and display a screen of the robotic processing automation apparatus including a remote terminal screen, the remote terminal screen including the received screen stream,
• transmit, when a user event occurs in the screen stream displayed in the screen of the robotic processing automation apparatus, an information inquiry request for an object related to the user event, the information inquiry request including coordinate information of the object on the screen of the remote apparatus, the coordinate information of the object being obtained based on coordinates of the object on the screen of the robotic processing automation apparatus,
• receive object-related information corresponding to the information inquiry request, and
• generate an automation script for automatically executing the target program based on the screen stream and the object-related information;
• and
• the remote apparatus configured to:
• convert the execution screen of the target program into a video stream in response to the screen stream request and transmit the converted screen stream to the robotic processing automation apparatus,
• receive the information inquiry request to detect an object corresponding to the information inquiry request, and
• extract object-related information on the object to transmit the extracted object-related information to the robotic processing automation apparatus,
• wherein the robotic processing automation apparatus is further configured to:
• extract first coordinates of the object from the screen of the robotic processing automation apparatus;
• convert the first coordinates of the object into second coordinates on the remote terminal screen based on a relative position between the screen of the robotic processing automation apparatus and the remote terminal screen;
• and
• convert the second coordinates into third coordinates on the screen of the remote apparatus using a resolution of the remote terminal screen and a resolution of the screen of the remote apparatus.