Learning method and learning device for removing jittering on video acquired through shaking camera by using a plurality of neural networks for fault tolerance and fluctuation robustness in extreme situations, and testing method and testing device using the same

A testing method for removing jittering in videos for testing generated by a shaking camera to remove the jittering on the videos for testing using one or more neural networks, comprising steps of:

• (a) a testing device, on condition that a learning device has performed processes of (1) after acquiring a t-th image for training corresponding to a t-th frame for training on videos for training, instructing a 1-st neural network to generate each of t-th masks for training corresponding to each of objects for training in the t-th image for training by applying at least one 1-st neural network operation to the t-th image for training, (2) instructing a 2-nd neural network to generate each of t-th object motion vectors for training of each of object pixels for training, corresponding to each of the objects for training, included in the t-th image for training by applying at least one 2-nd neural network operation to (i) each of the t-th masks for training, (ii) each of t-th cropped images for training, corresponding to each of the t-th masks for training, which are part of the t-th image for training, (iii) each of (t−1)-th masks for training, and (iv) each of (t−1)-th cropped images for training, corresponding to each of the (t−1)-th masks for training, which are part of a (t−1)-th image for training, (3) instructing a jittering estimation unit to generate each of t-th jittering vectors for training corresponding to each of reference pixels for training among pixels in the t-th image for training by referring to each of the t-th object motion vectors for training, and (4) instructing a learning unit to calculate one or more losses by referring to (i) t-th adjusted object motion vectors for training optimized through a process for generating the t-th jittering vectors for training and (ii) the t-th object motion vectors for training before the optimization, and to learn at least part of one or more parameters of the 2-nd neural network by backpropagating the losses;
• if a t-th image for testing corresponding to a t-th frame for testing on the videos for testing is acquired, instructing the 1-st neural network to generate each of t-th masks for testing corresponding to each of objects for testing in the t-th image for testing by applying the 1-st neural network operation to the t-th image for testing;
• (b) the testing device instructing the 2-nd neural network to generate each of t-th object motion vectors for testing of each of object pixels for testing, corresponding to each of the objects for testing, included in the t-th image for testing by applying the 2-nd neural network operation to (i) each of the t-th masks for testing, (ii) each of t-th cropped images for testing, corresponding to each of the t-th masks for testing, which are part of the t-th image for testing, (iii) each of (t−1)-th masks for testing, and (iv) each of (t−1)-th cropped images for testing, corresponding to each of the (t−1)-th masks for testing, which are part of a (t−1)-th image for testing;
• and
• (c) the testing device instructing the jittering estimation unit to generate each of t-th jittering vectors for testing corresponding to each of reference pixels for testing among pixels in the t-th image for testing by referring to each of the t-th object motion vectors for testing.

A computing device for detecting jittering in videos generated by a shaking camera to remove the jittering on the videos using one or more neural networks, comprising:

• at least one memory that stores instructions;
• and
• at least one processor configured to execute the instructions to:
• perform processes of:
• (I) if a t-th image corresponding to a t-th frame on the videos is acquired, instructing a 1-st neural network to generate each of t-th masks corresponding to each of objects in the t-th image by applying at least one 1-st neural network operation to the t-th image, (II) instructing a 2-nd neural network to generate each of t-th object motion vectors of each of object pixels, corresponding to each of the objects, included in the t-th image by applying at least one 2-nd neural network operation to (i) each of the t-th masks, (ii) each of t-th cropped images, corresponding to each of the t-th masks, which are part of the t-th image, (iii) each of (t−1)-th masks, and (iv) each of (t−1)-th cropped images, corresponding to each of the (t−1)-th masks, which are part of a (t−1)-th image, and (III) instructing a jittering estimation unit to generate each of t-th jittering vectors corresponding to each of reference pixels among pixels in the t-th image by referring to each of the t-th object motion vectors.

A testing device for removing jittering in videos for testing generated by a shaking camera to remove the jittering on the videos for testing using one or more neural networks, comprising:

• at least one memory that stores instructions;
• and
• at least one processor, on condition that a learning device has performed processes of (1) after acquiring a t-th image for training corresponding to a t-th frame for training on videos for training, instructing a 1-st neural network to generate each of t-th masks for training corresponding to each of objects for training in the t-th image for training by applying at least one 1-st neural network operation to the t-th image for training, (2) instructing a 2-nd neural network to generate each of t-th object motion vectors for training of each of object pixels for training, corresponding to each of the objects for training, included in the t-th image for training by applying at least one 2-nd neural network operation to (i) each of the t-th masks for training, (ii) each of t-th cropped images for training, corresponding to each of the t-th masks for training, which are part of the t-th image for training, (iii) each of (t−1)-th masks for training, and (iv) each of (t−1)-th cropped images for training, corresponding to each of the (t−1)-th masks for training, which are part of a (t−1)-th image for training, (3) instructing a jittering estimation unit to generate each of t-th jittering vectors for training corresponding to each of reference pixels for training among pixels in the t-th image for training by referring to each of the t-th object motion vectors for training, and (4) instructing a learning unit to calculate one or more losses by referring to (i) t-th adjusted object motion vectors for training optimized through a process for generating the t-th jittering vectors for training and (ii) the t-th object motion vectors for training before the optimization, and to learn at least part of one or more parameters of the 2-nd neural network by backpropagating the losses;
• configured to execute the instructions to:
• perform processes of:
• (I) if a t-th image for testing corresponding to a t-th frame for testing on the videos for testing is acquired, instructing the 1-st neural network to generate each of t-th masks for testing corresponding to each of objects for testing in the t-th image for testing by applying the 1-st neural network operation to the t-th image for testing, (II) instructing the 2-nd neural network to generate each of t-th object motion vectors for testing of each of object pixels for testing, corresponding to each of the objects for testing, included in the t-th image for testing by applying the 2-nd neural network operation to (i) each of the t-th masks for testing, (ii) each of t-th cropped images for testing, corresponding to each of the t-th masks for testing, which are part of the t-th image for testing, (iii) each of (t−1)-th masks for testing, and (iv) each of (t−1)-th cropped images for testing, corresponding to each of the (t−1)-th masks for testing, which are part of a (t−1)-th image for testing;
• and (III) instructing the jittering estimation unit to generate each of t-th jittering vectors for testing corresponding to each of reference pixels for testing among pixels in the t-th image for testing by referring to each of the t-th object motion vectors for testing.