Method and apparatus with image augmentation

A method with image augmentation, the method comprising:

• recognizing, based on a gaze of the user corresponding to an input image, any one or any combination of any two or more of an object of interest of a user, and a situation of the object of interest, from partial regions of the input image;
• recognizing a task being performed by the user by applying the partial regions to a second neural network continued to perform task recognition;
• determining relevant information indicating an intention of the user based on any two or any other combination of the object of interest of the user, the situation of the object of interest, and the task of the user;
• and
• generating a visually augmented image by visually augmenting the input image based on the relevant information,
• wherein the recognizing of the task comprises:
• generating an image sequence including the partial regions of the input image based on the gaze of the user,
• generating a coded image by visually encoding temporal information included in each gaze corresponding to the image sequence,
• extracting first feature vectors from partial regions to which gazes corresponding to the image sequence are mapped,
• extracting second feature vectors based on the coded image,
• obtaining feature vectors corresponding to the image sequence by concatenating the first feature vectors and the second feature vectors, and
• classifying the task based on the feature vectors.