Method for generating training data, and electronic device

An electronic device, comprising:

• at least one processor;
• and
• a memory communicatively coupled to the at least one processor and storing instructions executable by the at least one processor;
• wherein when the instructions are executed by the at least one processor, the at least one processor is caused to:
• obtain interaction data with an application, wherein the interaction data comprises a plurality of pieces of operation data, a pre-operation user interface image corresponding to each piece of operation data, and a post-operation user interface image corresponding to each piece of operation data;
• determine a state transition image based on the interaction data, wherein a node in the state transition image is the user interface image, and a connecting edge between every two user interface images is operation data associated with the two user interface images;
• generate a plurality of pieces of trajectory data based on the state transition image, wherein each piece of trajectory data indicates a trajectory image and an operation data sequence corresponding to a task;
• obtain reasoning reference information corresponding to the operation data in the trajectory data output by a multimodal model, by inputting the trajectory data into the multimodal model, wherein the reasoning reference information describes a reasoning process for the operation data;
• and
• generate training data for training an interaction agent model based on the trajectory data and the reasoning reference information corresponding to the operation data in the trajectory data.

A non-transitory computer readable storage medium, storing computer instructions, wherein the computer instructions are configured to enable a computer to perform a method for generating training data, the method comprising:

• obtaining interaction data with an application, wherein the interaction data comprises a plurality of pieces of operation data, a pre-operation user interface image corresponding to each piece of operation data, and a post-operation user interface image corresponding to each piece of operation data;
• determining a state transition image based on the interaction data, wherein a node in the state transition image is the user interface image, and a connecting edge between every two user interface images is operation data associated with the two user interface images;
• generating a plurality of pieces of trajectory data based on the state transition image, wherein each piece of trajectory data indicates a trajectory image and an operation data sequence corresponding to a task;
• obtaining reasoning reference information corresponding to the operation data in the trajectory data output by the multimodal model, by inputting the trajectory data into a multimodal model, wherein the reasoning reference information describes a reasoning process for the operation data;
• and
• generating training data for training an interaction agent model based on the trajectory data and the reasoning reference information corresponding to the operation data in the trajectory data.