Learning method and learning device for generating training data from virtual data on virtual world by using generative adversarial network, to thereby reduce annotation cost required in training processes of neural network for autonomous driving, and a testing method and a testing device using the same

A testing method for transforming a virtual video on a virtual world to a more real-looking video whose characteristics are similar to those of a real video on a real world while maintaining constant and uniform relationships among frames thereof by using information on previous frames when transforming a current frame, comprising steps of:

• (a) on condition that (1) a learning device, implemented as a programmed computer, under a condition of (i) an N-th virtual training image corresponding to an N- th frame for training of a virtual video for training on the virtual world, (ii) N-th meta data for training on one or more objects for training included in the N-th virtual training image, and (iii) (N-K)-th reference information for training on an (N-K)-th frame for training of the virtual video for training are acquired, has instructed at least one encoding layer in a generating Convolutional Neural Network(CNN) to apply at least one convolutional operation to the N-th virtual training image, the N-th meta data for training and the (N-K)-th reference information for training, to thereby generate at least one N-th feature map for training;
• (2) the learning device has instructed a decoding layer in the generating CNN to apply at least one deconvolutional operation to the at least one N-th feature map for training, to thereby generate an N-th transformed image for training;
• (3) the learning device has instructed a discriminating CNN, which is trained adversely to the generating CNN, to apply at least one discriminating CNN operation to the N-th transformed image for training, to thereby generate a first estimated image category score vector including information on probabilities of the N-th transformed image for training being one of real images on the real world or one of virtual images on the virtual world;
• (4) the learning device has instructed a generating loss layer in the generating CNN to generate at least one generating CNN loss by referring to the first estimated image category score vector and its corresponding first Ground-Truth(GT) image category score vector, and then to perform backpropagation by referring to the generating CNN loss, to thereby learn at least some parameters of the generating CNN, wherein the learning device has instructed the generating loss layer to train said at least some parameters of the generating CNN in order to make a probability of the discriminating CNN misdetermining a category of an image to be outputted from the generating CNN larger, at (1), the learning device instructing at least one adjusting laver in the generating CNN to apply at least one adjusting operation to the at least one N-th feature map by referring to an N-th random seed value, to thereby generate at least one N-th adjusted feature map, at (2), the learning device instructing the decoding laver to apply the deconvolutional operation to the at least one N-th adjusted feature map, to thereby generate the N- th transformed image, and repeating (1), (2), (3) and (4) for each frame of the virtual video on the virtual world, wherein each random seed value with its corresponding different index is separately and independently generated, a testing device, implemented with the programmed computer, under a condition of (i) an N-th virtual testing image corresponding to an N-th frame for testing of a virtual video for testing on the virtual world, (ii) N-th meta data for testing on one or more objects for testing included in the N-th virtual testing image, and (iii) (N-K)-th reference information for testing on an (N-K)-th frame for testing of the virtual video for testing are acquired, instructing the encoding layer in the generating CNN to apply the convolutional operation to the N-th virtual testing image, the N-th meta data for testing and the (N-K)-th reference information for testing, to thereby generate at least one N-th feature map for testing;
• and (b) the testing device instructing the decoding layer in the generating CNN to apply the deconvolutional operation to the at least one N-th feature map for testing, to thereby generate an N-th transformed image for testing.

A learning device for transforming a virtual video on a virtual world to a more real-looking video whose characteristics are similar to those of a real video on a real world while maintaining constant and uniform relationships among frames thereof by using information on previous frames when transforming a current frame, comprising:

• at least one memory that stores instructions:
• and
• at least one processor configured to execute the instructions to perform processes of:
• (I) if (i) an N-th virtual training image corresponding to an N-th frame of the virtual video on the virtual world, (ii) N-th meta data on one or more objects included in the N-th virtual training image, and (iii) (N-K)-th reference information on an (N-K)-th frame of the virtual video are acquired, instructing at least one encoding layer in a generating Convolutional Neural Network(CNN) to apply at least one convolutional operation to the N-th virtual training image, the N-th meta data and the (N-K)-th reference information, to thereby generate at least one N-th feature map;
• (II) instructing a decoding layer in the generating CNN to apply at least one deconvolutional operation to the at least one N-th feature map, to thereby generate an N-th transformed image;
• (Ill) instructing a discriminating CNN, which is trained adversely to the generating CNN, to apply at least one discriminating CNN operation to the N-th transformed image, to thereby generate a first estimated image category score vector including information on probabilities of the N-th transformed image being one of real images on the real world or one of virtual images on the virtual world;
• (IV) instructing a generating loss layer in the generating CNN to generate at least one generating CNN loss by referring to the first estimated image category score vector and its corresponding first Ground-Truth(GT) image category score vector, and then to perform backpropagation by referring to the generating CNN loss, to thereby learn at least some parameters of the generating CNN, wherein the processor instructs the generating loss layer to train said at least some parameters of the generating CNN in order to make a probability of the discriminating CNN misdetermining a category of an image to be outputted from the generating CNN larger, wherein
• at the process of (I), the processor instructs at least one adjusting laver in the generating CNN to apply at least one adjusting operation to the at least one N-th feature map by referring to an N-th random seed value, to thereby generate at least one N-th adjusted feature map, and
• wherein, at the process of (II), the processor instructs the decoding laver to apply the deconvolutional operation to the at least one N-th adjusted feature map, to thereby generate the N-th transformed image, and the processor repeats (I), (II), (Ill), and (IV) for each frame of the virtual video on the virtual world, wherein each random seed value with its corresponding different index is separately and independently generated.

A testing device for transforming a virtual video on a virtual world to a more real-looking video whose characteristics are similar to those of a real video on a real world while maintaining constant and uniform relationships among frames thereof by using information on previous frames when transforming a current frame, comprising:

• at least one memory that stores instructions;
• and
• at least one processor configured to execute the instructions to perform processes of:
• (I) on condition that (1) a learning device, under a condition of (i) an N-th virtual training image corresponding to an N-th frame for training of a virtual video for training on the virtual world, (ii) N-th meta data for training on one or more objects for training included in the N-th virtual training image, and (iii) (N-K)-th reference information for training on an (N-K)-th frame for training of the virtual video for training are acquired, has instructed at least one encoding layer in a generating Convolutional Neural Network(CNN) to apply at least one convolutional operation to the N-th virtual training image, the N-th meta data for training and the (N-K)-th reference information for training, to thereby generate at least one N-th feature map for training;
• (2) the learning device has instructed a decoding layer in the generating CNN to apply at least one deconvolutional operation to the at least one N-th feature map for training, to thereby generate an N-th transformed image for training;
• (3) the learning device has instructed a discriminating CNN, which is trained adversely to the generating CNN, to apply at least one discriminating CNN operation to the N-th transformed image for training, to thereby generate a first estimated image category score vector including information on probabilities of the N-th transformed image for training being one of real images on the real world or one of virtual images on the virtual world;
• (4) the learning device has instructed a generating loss layer in the generating CNN to generate at least one generating CNN loss by referring to the first estimated image category score vector and its corresponding first Ground-Truth(GT) image category score vector, and then to perform backpropagation by referring to the generating CNN loss, to thereby learn at least some parameters of the generating CNN, wherein the learning device has instructed the generating loss layer to train said at least some parameters of the generating CNN in order to make a probability of the discriminating CNN misdetermining a category of an image to be outputted from the generating CNN larger, wherein
• at the process of (1), the processor instructs at least one adjusting laver in the generating CNN to apply at least one adjusting operation to the at least one N-th feature map by referring to an N-th random seed value, to thereby generate at least one N-th adjusted feature map, and at the process of (2), the processor instructs the decoding laver to apply the deconvolutional operation to the at least one N-th adjusted feature map, to thereby generate the N-th transformed image, and the processor repeats (1), (2), (3), and (4) for each of the virtual video on the virtual world, wherein each random seed value with its corresponding different index is separately and independently generated if under a condition of (i) an N-th virtual testing image corresponding to an N-th frame for testing of a virtual video for testing on the virtual world, (ii) N-th meta data for testing on one or more objects for testing included in the N-th virtual testing image, and (iii) (N-K)-th reference information for testing on an (N-K)-th frame for testing of the virtual video for testing are acquired, instructing the encoding layer in the generating CNN to apply the convolutional operation to the N-th virtual testing image, the N-th meta data for testing and the (N-K)-th reference information for testing, to thereby generate at least one N-th feature map for testing;
• and (II) instructing the decoding layer in the generating CNN to apply the deconvolutional operation to the at least one N-th feature map for testing, to thereby generate an N-th transformed image for testing.