Method and system for identifying grid format object in robotic process automation environment

A system for automating work processing by traversing items in a grid-format structure using a robotic process automation (RPA) technology, by identifying a user interface (UI) object displayed on a screen using a non-grid-format UI control in an RPA environment, as a grid-format object, the system comprising:

• a communication interface;
• a memory configured to load thereon a computer program;
• and
• at least one processor configured to execute the computer program,
• wherein the computer program includes instructions for performing:
• an operation of receiving, via a user interface (UI), a search reference object, which is an UI object designated by a user and displayed using a non-grid-format UI control of a target work program, wherein the search reference object is visually expressed as a grid-format UI structure on a screen of the target work program, and the non-grid-format UI control does not provide information about each of rows and each of columns in the grid-format UI structure;
• an operation of providing, by an operating system, automation characteristic information including a position, a size, and a control type of the UI object expressed by a node of a hierarchical structure tree;
• an operation of searching for an ancestor object of the search reference object on the hierarchical structure tree and determining the searched ancestor object as a first row candidate object using the automation characteristic information provided by the operating system, wherein the determining of the searched ancestor object as the first row candidate object includes;
• (i) determining whether an application of the search reference object and an application of a parent object of the search reference object are identical with each other based on whether the search reference object and the parent object are grid-format information displayed on the same UI screen and are objects belonging to the same program;
• and
• (ii) upon determination that the application of the search reference object and the application of the parent object of the search reference object are identical with each other, determining the search reference object as the first row candidate object;
• an operation of determining a sibling object of the first row candidate object on the hierarchical structure tree as a second row candidate object, by using automation characteristic information of the first row candidate object and automation characteristic information of the sibling object of the first row candidate object, wherein the determining of the sibling object of the first row candidate object as the second row candidate object includes:
• (iii) determining whether the automation characteristic information of the sibling object of the first row candidate object and the automation characteristic information of the first row candidate object are identical with each other;
• and
• (iv) comparing a structure of a first sibling object having the automation characteristic information identical with the automation characteristic information of the first row candidate object with a structure of the first row candidate object;
• an operation of determining whether a child object of the first row candidate object and a child object of the second row candidate object satisfy a column relationship with each other on the hierarchical structure tree;
• an operation of identifying each of the first row candidate object and the second row candidate object as a grid-format object, based on a result of determination on the column relationship;
• and
• an operation of creating an automation scenario that performs automated work processing by traversing items of the grid-format object,
• wherein steps (i)-(iv) are recursively performed until a pair of the first row candidate object and the second row candidate object having the same automation characteristic information and the same structure has been detected.

A non-transitory computer readable medium storing a computer program and connected to a computer device, wherein when the computer program is executed by a computing device, the computer program causes the computer device to perform a method for automating work processing by traversing items in a grid-format structure using a robotic process automation (RPA) technology, by identifying a user interface (UI) object displayed on a screen using a non-grid-format UI control in an RPA environment, as a grid-format object, the method comprising:

• an operation of receiving, via a user interface (UI), a search reference object, which is an UI object designated by a user and displayed using a non-grid-format UI control of a target work program, wherein the search reference object is visually expressed as a grid-format UI structure on a screen of the target work program, and the non-grid-format UI control does not provide information about each of rows and each of columns in the grid-format UI structure;
• an operation of providing, by an operating system, automation characteristic information including a position, a size, and a control type of the UI object expressed by a node of a hierarchical structure tree;
• an operation of searching for an ancestor object of the search reference object on the hierarchical structure tree and determining the searched ancestor object as a first row candidate object using the automation characteristic information provided by the operating system, wherein the determining of the searched ancestor object as the first row candidate object includes:
• (i) determining whether an application of the search reference object and an application of a parent object of the search reference object are identical with each other based on whether the search reference object and the parent object are grid-format information displayed on the same UI screen and are objects belonging to the same program;
• and
• (ii) upon determination that the application of the search reference object and the application of the parent object of the search reference object are identical with each other, determining the search reference object as the first row candidate object;
• an operation of determining a sibling object of the first row candidate object on the hierarchical structure tree as a second row candidate object, by using automation characteristic information of the first row candidate object and automation characteristic information of the sibling object of the first row candidate object, wherein the determining of the sibling object of the first row candidate object as the second row candidate object includes:
• (iii) determining whether the automation characteristic information of the sibling object of the first row candidate object and the automation characteristic information of the first row candidate object are identical with each other;
• and
• (iv) comparing a structure of a first sibling object having the automation characteristic information identical with the automation characteristic information of the first row candidate object with a structure of the first row candidate object;
• an operation of determining whether a child object of the first row candidate object and a child object of the second row candidate object satisfy a column relationship with each other on the hierarchical structure tree;
• an operation of identifying each of the first row candidate object and the second row candidate object as a grid-format object, based on a result of determination on the column relationship;
• and
• an operation of creating an automation scenario that performs automated work processing by traversing items of the grid-format object,
• wherein steps (i)-(iv) are recursively performed until a pair of the first row candidate object and the second row candidate object having the same automation characteristic information and the same structure has been detected.