# US 10,311,335 B1 · Method and device for generating image data set to be used for learning CNN capable of detecting obstruction in autonomous driving circumstance, and testing method, and testing device using the same > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,311,335 B1 - Document ID: US10311335B1 - Status: Granted - Kind: B1 - Application number: 16/121,635 - Filed: 2018-09-05 - Granted: 2019-06-04 - Earliest claimed: 2018-09-05 - Assignees: StradVision, Inc. - Applicants: Stradvision, Inc. - Inventors: Kye-Hyeon Kim; Yongjoong Kim; Insu Kim; Hak-Kyoung Kim; Woonhyun Nam; SukHoon Boo; Myungchul Sung; Donghun Yeo; Wooju Ryu; Taewoong Jang; Kyungjoong Jeong; Hongmo Je; Hojin Cho - Web page: https://patentgrep.com/patent/US10311335B1 ## CPC - G06N 3/084: Backpropagation, e.g. using gradient descent - G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00) - G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade} - G06F 18/24143: {Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]} - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0475: Generative networks - G06N 3/09: Supervised learning - G06N 3/094: Adversarial learning - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/764: using classification, e.g. of video objects - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/82: using neural networks - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method of generating at least one image data set to be used for learning CNN capable of detecting at least one obstruction in one or more autonomous driving circumstances, comprising steps of: (a) a learning device acquiring (i) an original image representing a road driving circumstance and (ii) a synthesized label obtained by using an original label corresponding to the original image and an additional label corresponding to an arbitrary specific object, wherein the arbitrary specific object does not relate to the original image; and (b) the learning device supporting a first CNN module to generate a synthesized image using the original image and the synthesized label, wherein the synthesized image is created by combining (i) an image of the arbitrary specific object corresponding to the additional label and (ii) the original image. ## Claims ### Claim 1 A method of generating at least one image data set to be used for learning CNN capable of detecting at least one obstruction in one or more autonomous driving circumstances, comprising steps of: (a) a learning device acquiring (i) an original image representing a road driving circumstance and (ii) a synthesized label obtained by using an original label corresponding to the original image and an additional label corresponding to an arbitrary specific object, wherein the arbitrary specific object does not relate to the original image; (b) the learning device supporting a first CNN module to generate a synthesized image using the original image and the synthesized label, wherein the synthesized image is created by combining (i) an image of the arbitrary specific object corresponding to the additional label and (ii) the original image; (c) the learning device supporting a second CNN module to acquire the synthesized image and to generate a first output value which represents a probability as to whether the synthesized image is fake or real, and further supporting the second CNN module to acquire an arbitrary real image which has not been synthesized, and to generate a second output value which represents a probability as to whether the arbitrary real image is fake or real; and (d) the learning device supporting the second CNN module to generate a first loss value by comparing the first output value to a first GT value, and a second loss value by comparing the second output value to a second GT value, and supporting the first and the second CNN modules to be learned by backpropagating each of the first and the second loss values through the second CNN module and the first CNN module. ### Claim 2 (depends on 1) The method of claim 1, wherein, at the step of (d), the learning device supports the first CNN module to learn parameters thereof such that a degree of making the second CNN module judge the synthesized image as real image is increased, and the second CNN module to learn parameters thereof such that a degree of correctly judging each of the arbitrary real image and synthesized image is whether real or fake is increased. ### Claim 3 (depends on 1) The method of claim 1, wherein, at the step of (a), the learning device further acquires (iii) at least a value of a random seed, wherein, at the step of (b), the learning device supports the first CNN module to generate one or more adjusted synthesized images using the original image, the synthesized label and the value of the random seed, and wherein each of the adjusted synthesized images is derived from the synthesized image such that at least part of a size, a location, and a color of the specific object in each of the adjusted synthesized images are adjusted from those of the specific object in the synthesized image while changing the value of the random seed. ### Claim 4 (depends on 3) The method of claim 3, wherein, the first CNN module includes; an encoding layer for generating a feature map by applying one or more convolution operations to the original image and the synthesized label, an adjusting layer for generating an adjusted feature map by using the feature map and the value of the random seed, and a decoding layer for generating the synthesized image by applying one or more deconvolution operations to the adjusted feature map. ### Claim 5 (depends on 1) The method of claim 1, wherein, at the step of (a), at the time when the original image and the synthesized label are acquired, the learning device further acquires (i) information on one or more first bounding boxes corresponding to one or more objects included in the original image and (ii) information on a second bounding box corresponding to the specific object included in the synthesized label, wherein, at the step of (b), the learning device further generates third bounding boxes on the synthesized image by using the information on the first and the second bounding boxes, and wherein the third bounding boxes include at least one of boxes corresponding to (i) at least one of the multiple objects in the original image and (ii) the specific object in the synthesized label, or adjusted boxes that have been slightly resized or relocated based on the boxes corresponding to (i) and (ii). ### Claim 6 (depends on 5) The method of claim 5, wherein the second bounding box is generated by using minimum axis parallel rectangle. ### Claim 7 (depends on 1) The method of claim 1, wherein, at the step of (a), after a type of the specific object is selected among a plurality of data sets, the additional label is generated by sampling multiple masks corresponding to the type of the specific object. ### Claim 8 (depends on 1) The method of claim 1, further comprising a step of: e) the learning device adding the synthesized image and the synthesized label into a database including training data sets for one of object detection and image segmentation. ### Claim 9 A test method of generating at least one image data set to be used for detecting at least one obstruction in one or more autonomous driving circumstances, comprising: (a) a testing device, on condition that (I) a learning device has acquired (i) an original image for training representing a road driving circumstance and (ii) a synthesized label for training obtained by using an original label for training corresponding to the original image for training and an additional label for training corresponding to an arbitrary specific object for training, wherein the arbitrary specific object for training does not relate to the original image for training; (II) the learning device has supported a first CNN module to generate a synthesized image for training using the original image for training and the synthesized label for training, wherein the synthesized image for training is created by combining (i) an image of the arbitrary specific object for training corresponding to the additional label for training and (ii) the original image for training, (III) the learning device has supported, a second CNN module to acquire the synthesized image for training and to generate a first output value which represents a probability as to whether the synthesized image for training is fake or real, and further has supported the second CNN module to acquire an arbitrary real image which has not been synthesized, and to generate a second output value which represents a probability as to whether the arbitrary real image is fake or real; and (IV) the learning device has supported the second CNN module to generate a first loss value by comparing the first output value to a first GT value, and a second loss value by comparing the second output value to a second GT value, and has supported the first and the second CNN modules to be learned by backpropagating each of the first and the second loss values through the second CNN module and the first CNN module, acquiring (i) an original image for testing representing a road driving circumstance and (ii) a synthesized label for testing obtained by using an original label for testing corresponding to the original image for testing and an additional label for testing corresponding to an arbitrary specific object for testing, wherein the arbitrary specific object for testing does not relate to the original image for testing; and (b) the testing device supporting the first CNN module to generate a synthesized image for testing using the original image for testing and the synthesized label for testing, wherein the synthesized image for testing is created by combining (i) an image of the arbitrary specific object for testing corresponding to the additional label for testing and (ii) the original image for testing. ### Claim 10 (depends on 9) The method of claim 9, wherein, at the step of (a), the testing device further acquires (iii) at least a value of a random seed for testing, wherein, at the step of (b), the testing device supports the first CNN module to generate one or more adjusted synthesized images for testing using the original image for testing, the synthesized label for testing and the value of the random seed for testing, and wherein each of the adjusted synthesized images for testing is derived from the synthesized image for testing such that at least part of a size, a location, and a color of the specific object for testing in each of the adjusted synthesized images for testing are adjusted from those of the specific object for testing in the synthesized image for testing while changing the value of the random seed for testing. ### Claim 11 (depends on 9) The method of claim 9, wherein, the first CNN module includes: an encoding layer for generating a feature map for testing by applying one or more convolution operations to the original image for testing and the synthesized label for testing, an adjusting layer for generating an adjusted feature map for testing by using the feature map for testing and a value of a random seed for testing, and a decoding layer for generating the synthesized image for testing by applying one or more deconvolution operations to the adjusted feature map for testing. ### Claim 12 (depends on 9) The method of claim 9, wherein, at the step of (a), at the time when the original image for testing and the synthesized label for testing are acquired, the testing device further acquires (i) information on one or more first bounding boxes for testing corresponding to one or more objects included in the original image for testing and (ii) information on a second bounding box for testing corresponding to the specific object for testing included in the synthesized label for testing, wherein, at the step of (b), the learning device further generates third bounding boxes for testing on the synthesized image for testing by using the information on the first and the second bounding boxes for testing, and wherein the third bounding boxes for testing include at least one of boxes corresponding to (i) at least one of the multiple objects in the original image for testing and (ii) the specific object for testing in the synthesized label for testing, or adjusted boxes for testing that have been slightly resized or relocated based on the boxes corresponding to (i) and (ii). ### Claim 13 (depends on 12) The method of claim 12, wherein the second bounding box for testing is generated by using minimum axis parallel rectangle. ### Claim 14 (depends on 9) The method of claim 9, wherein, at the step of (a), after a type of the specific object for testing is selected among a plurality of data sets, the additional label for testing is generated by sampling multiple masks corresponding to the type of the specific object for training. ### Claim 15 A device of generating at least one image data set to be used for learning CNN capable of detecting at least one obstruction in one or more autonomous driving circumstances, comprising: a communication part for acquiring (i) an original image representing a road driving circumstance and (ii) a synthesized label obtained by using an original label corresponding to the original image and an additional label corresponding to an arbitrary specific object, wherein the arbitrary specific object does not relate to the original image; a processor for performing a process of (I) supporting a first CNN module to generate a synthesized image using the original image and the synthesized label, wherein the synthesized image is created by combining (i) an image of the arbitrary specific object corresponding to the additional label and (ii) the original image; (II) supporting a second CNN module to acquire the synthesized image and to generate a first output value which represents a probability as to whether the synthesized image is fake or real, and supporting the second CNN module to acquire an arbitrary real image which has not been synthesized, and to generate a second output value which represents a probability as to whether the arbitrary real image is fake or real; and (III) supporting the second CNN module to generate a first loss value by comparing the first output value to a first GT value, and a second loss value by comparing the second output value to a second GT value, and supporting the first and the second CNN modules to be learned by backpropagating each of the first and the second loss values through the second CNN module and the first CNN module. ### Claim 16 (depends on 15) The device of claim 15, wherein, at the process of (III), the processor supports the first CNN module to learn parameters thereof such that a degree of making the second CNN module judge the synthesized image as real image is increased, and the processor supports the second CNN module to learn parameters thereof such that a degree of correctly judging each of the arbitrary real image and synthesized image is whether real or fake is increased. ### Claim 17 (depends on 15) The device of claim 15, wherein the communication part further acquires (iii) at least a value of a random seed, wherein, at the process of (I), the processor supports the first CNN module to generate one or more adjusted synthesized images using the original image, the synthesized label and the value of the random seed, and wherein each of the adjusted synthesized images is derived from the synthesized image such that at least part of a size, a location, and a color of the specific object in each of the adjusted synthesized images are adjusted from those of the specific object in the synthesized image while changing the value of the random seed. ### Claim 18 (depends on 17) The device of claim 17, wherein the first CNN module includes: an encoding layer for generating a feature map by applying one or more convolution operations to the original image and the synthesized label, an adjusting layer for generating an adjusted feature map by using the feature map and the value of the random seed, and a decoding layer for generating the synthesized image by applying one or more deconvolution operations to the adjusted feature map. ### Claim 19 (depends on 15) The device of claim 15, wherein, at the time when the original image and the synthesized label are acquired, the communication part further acquires (i) information on one or more first bounding boxes corresponding to one or more objects included in the original image and (ii) information on a second bounding box corresponding to the specific object included in the synthesized label, wherein, at the process of (I), the processor further generates third bounding boxes on the synthesized image by using the information on the first and the second bounding boxes, and wherein the third bounding boxes include at least one of boxes corresponding to (i) at least one of the multiple objects in the original image and (ii) the specific object in the synthesized label, or adjusted boxes that have been slightly resized or relocated based on the boxes corresponding to (i) and (ii). ### Claim 20 (depends on 19) The device of claim 19, wherein the second bounding box is generated by using minimum axis parallel rectangle. ### Claim 21 (depends on 15) The device of claim 15, wherein, after a type of the specific object is selected among a plurality of data sets, the additional label is generated by sampling multiple masks corresponding to the type of the specific object. ### Claim 22 (depends on 15) The device of claim 15, wherein the processor further performs a process of: (IV) adding the synthesized image and the synthesized label into a database including training data sets for one of object detection and image segmentation. ### Claim 23 A test device of generating at least one image data set to be used for detecting at least one obstruction in one or more autonomous driving circumstances, comprising: a communication part, on condition that (I) a learning device has acquired (i) an original image for training representing a road driving circumstance and (ii) a synthesized label for training obtained by using an original label for training corresponding to the original image for training and an additional label for training corresponding to an arbitrary specific object for training, wherein the arbitrary specific object for training does not relate to the original image for training; (II) the learning device has supported a first CNN module to generate a synthesized image for training using the original image for training and the synthesized label for training, wherein the synthesized image for training is created by combining (i) an image of the arbitrary specific object for training corresponding to the additional label for training and (ii) the original image for training, (III) the learning device has supported, a second CNN module to acquire the synthesized image for training and to generate a first output value which represents a probability as to whether the synthesized image for training is fake or real, and further has supported the second CNN module to acquire an arbitrary real image which has not been synthesized, and to generate a second output value which represents a probability as to whether the arbitrary real image is fake or real; and (IV) the learning device has supported the second CNN module to generate a first loss value by comparing the first output value to a first GT value, and a second loss value by comparing the second output value to a second GT value, and has supported the first and the second CNN modules to be learned by backpropagating each of the first and the second loss values through the second CNN module and the first CNN module, for acquiring (i) an original image for testing representing a road driving circumstance and (ii) a synthesized label for testing obtained by using an original label for testing corresponding to the original image for testing and an additional label for testing corresponding to an arbitrary specific object for testing, wherein the arbitrary specific object for testing does not relate to the original image for testing; and a processor for performing a process of (I) supporting the first CNN module to generate a synthesized image for testing using the original image for testing and the synthesized label for testing, wherein the synthesized image for testing is created by combining (i) an image of the arbitrary specific object for testing corresponding to the additional label for testing and (ii) the original image for testing. ### Claim 24 (depends on 23) The device of claim 23, wherein the communication part further acquires (iii) at least a value of a random seed value for testing, wherein, at the process of (I), supports the first CNN module to generate one or more adjusted synthesized images for testing using the original image for testing, the synthesized label for testing and the value of the random seed for testing, and wherein each of the adjusted synthesized images for testing is derived from the synthesized image for testing such that at least part of a size, a location, and a color of the specific object for testing in each of the adjusted synthesized images for testing are adjusted from those of the specific object for testing in the synthesized image for testing while changing the value of the random seed for testing. ### Claim 25 (depends on 23) The device of claim 23, wherein the first CNN module includes: an encoding layer for generating a feature map for testing by applying one or more convolution operations to the original image for testing and the synthesized label for testing, an adjusting layer for generating an adjusted feature map for testing by using the feature map for testing and the value of the random seed for testing, and a decoding layer for generating the synthesized image for testing by applying one or more deconvolution operations to the adjusted feature map for testing. ### Claim 26 (depends on 23) The device of claims 23, wherein, at the time when the original image for testing and the synthesized label for testing are acquired, the communication part further acquires (i) information on one or more first bounding boxes for testing corresponding to one or more objects included in the original image for testing and (ii) information on a second bounding box for testing corresponding to the specific object for testing included in the synthesized label for testing, wherein, at the process of (I), the learning device further generates third bounding boxes for testing on the synthesized image for testing by using the information on the first and the second bounding boxes for testing, and wherein the third bounding boxes for testing include at least one of boxes corresponding to (i) at least one of the multiple objects in the original image for testing and (ii) the specific object for testing in the synthesized label for testing, or adjusted boxes for testing that have been slightly resized or relocated based on the boxes corresponding to (i) and (ii). ### Claim 27 (depends on 26) The device of claim 26, wherein the second bounding box for testing is generated by using minimum axis parallel rectangle. ### Claim 28 (depends on 23) The device of claim 23, wherein, after a type of the specific object for testing is selected among a plurality of data sets, the additional label for testing is generated by sampling multiple masks corresponding to the type of the specific object for training.