Device for generating prediction image on basis of generator including concentration layer, and control method therefor

A method for controlling an electronic apparatus including a memory storing a generator previously trained to generate a prediction image based on one or more inputted images is stored,

• wherein the generator is configured to include a first neural network for performing encoding with respect to a plurality of inputted image frames, and a second neural network which is connected with the first neural network and configured to perform decoding with respect to data encoded through the first neural network,
• wherein the first neural network includes a first attention layer and the second neural network includes a second attention layer, the method comprising:
• inputting a plurality of image frames inputted according to time into the generator;
• acquiring feature data from a plurality of image frames input through at least one layer included in the generator,
• extracting feature data corresponding to change over time from the feature data acquired through an attention layer included in the generator,
• acquiring a first image data block by performing max-pooling and deconvolution through the first attention layer,
• acquiring a second image data block by performing max-pooling and deconvolution through the second attention layer,
• acquiring a third image data block by performing concatenation of connecting the first image data block to the second image data block by inputting the first image data block in the second attention layer,
• wherein the third image data block includes feature data for a resolution which is smaller than the existing number of pixels and image data corresponding to a change in motion over time,
• acquiring a first prediction image frame by inputting the extracted feature data and the third image data block to at least one other layer included in the generator,
• wherein each of the plurality of image frames consists of a plurality of pixels, and
• based on a result of comparing an image frame inputted after the plurality of image frames and the first prediction image frame, training the generator to extract feature data of pixels predicted to change over time from feature data for each of the plurality of pixels outputted from the at least one layer, and
• acquiring a second prediction image frame based on the trained generator.

A non-transitory computer-readable recording medium that is executed by a processor of an electronic apparatus including a memory storing a generator previously trained to generate a prediction image based on one or more inputted image,

• wherein the generator is configured to include a first neural network for performing encoding with respect to a plurality of inputted image frames, and a second neural network which is connected with the first neural network and configured to perform decoding with respect to data encoded through the first neural network,
• wherein the first neural network includes a first attention layer and the second neural network includes a second attention layer,
• the non-transitory computer-readable recording medium storing at least one instruction to cause the electronic apparatus to perform a plurality of operations comprising:
• inputting a plurality of image frames according to time;
• acquiring feature data from the plurality of image frames input through at least one layer included in the generator;
• extracting feature data corresponding to change over time from the feature data acquired through an attention layer included in the generator,
• acquiring a first image data block by performing max-pooling and deconvolution through the first attention layer,
• acquiring a second image data block by performing max-pooling and deconvolution through the second attention layer,
• acquiring a third image data block by performing concatenation of connecting the first image data block to the second image data block by inputting the first image data block in the second attention layer,
• wherein the third image data block includes feature data for a resolution which is smaller than the existing number of pixels and image data corresponding to a change in motion over time,
• acquiring a first prediction image frame by inputting the extracted feature data and the third image data block to at least one other layer included in the generator,
• wherein each of the plurality of image frames consists of a plurality of pixels, and
• based on a result of comparing an image frame inputted after the plurality of image frames and the first prediction image frame, training the generator to extract feature data of pixels predicted to change over time from feature data for each of the plurality of pixels outputted from the at least one layer, and
• acquiring a second prediction image frame based on the trained generator.