Information processing method for displaying of graphical user interface screen using trained model

An information processing method implementable with a computer, the method comprising:

• executing a target program to cause displaying of a graphical user interface screen;
• detecting an object being operable and included in the graphical user interface screen displayed by execution of the target program, using a trained model being trained to output, in response to an input of an image of a graphical user interface screen, information about an object included in the graphical user interface screen,
• wherein the trained model is generated using training data including an image of a graphical user interface screen and information about an object included in the graphical user interface screen, in which different designs associated with a same processing instruction are trained to be recognized as a same type of an object;
• obtaining data about transition of a screen, wherein the transition of the screen is to be displayed in response to an operation on the detected object;
• and
• generating screen transition data for the target program by associating the graphical user interface screen before transition, a type of the object in the graphical user interface screen, and the data about transition of the screen to be displayed in response to the operation on the object with one another,
• wherein:
• the object is assigned with a priority level;
• the storage medium storing the program further causes the computer to perform operations comprising generating, upon detecting a plurality of objects in the graphical user interface screen, screen transition data by obtaining, in order of priority from an object with a highest priority level, data about transition of the screen to be displayed in response to an operation on each of the plurality of objects;
• the graphical user interface screen has a plurality of hierarchical levels;
• and
• the priority level is assigned to each of the plurality of objects based on whether screen transition data for a screen is generated by causing transition of the screen in a depth direction of the plurality of hierarchical levels of the graphical user interface screen or by causing transition to a screen at a same depth in the plurality of hierarchical levels for each depth of the plurality of hierarchical levels.

A non-transitory computer readable storage medium storing a program for causing a computer to perform operations comprising:

• executing a target program to cause displaying of a graphical user interface screen;
• detecting an object being operable and included in the graphical user interface screen displayed by execution of the target program, using a trained model being trained to output, in response to an input of an image of a graphical user interface screen, information about an object included in the graphical user interface screen,
• wherein the trained model is generated using training data including an image of a graphical user interface screen and information about an object included in the graphical user interface screen, in which different designs associated with a same processing instruction are trained to be recognized as a same type of an object;
• obtaining data about transition of a screen, wherein the transition of the screen is to be displayed in response to an operation on the detected object;
• identifying data about transition of the screen associated with the graphical user interface screen before transition and a type of the object from the screen transition data generated before the target program or the execution environment of the target program is updated;
• comparing the obtained data about transition of the screen with the identified data about transition of the screen;
• and
• outputting a result of comparison,
• wherein the operations further comprises:
• calculating a degree of similarity between the obtained data about transition of the screen and the identified data about transition of the screen;
• and
• outputting the graphical user interface screen before transition and the type of the object if the calculated degree of similarity is less than a predetermined value.