Using deep video frame prediction for training a controller of an autonomous vehicle

A system comprising one or more processing devices and one or more memory devices operably coupled to the one or more processing devices, the one or more memory devices storing executable code effective to cause the one or more processing devices to:

• for each set of images of sets of images, input the each set of images to a control generator and receive a corresponding control input for the each set of images from the control generator in response to the each set of images;
• for each set of images of the set of images, input the each set of images to an image predictor along with the corresponding control input for the each set of images, the image predictor programmed to output a predicted image for the each set of images and the corresponding control input for the each set of images;
• for each set of images of the sets of images, input the each set of images and the predicted image for the each set of images into a discriminator;
• and
• use an output from the discriminator for each set of images as feedback to the control generator.

A method comprising, by a computing device:

• receiving a first image stream from a first camera mounted to a first vehicle, the first image stream captured in a first time period;
• receiving a first control input stream for the first vehicle, the first control input stream captured during the first time period and including inputs received from a driver of the first vehicle;
• generating first training data sets, each training data set including
• N contiguous images from the first image stream, where N is an integer greater than one,
• one or more control inputs from the first control input stream corresponding temporally to the first image stream, and
• an image immediately following the N contiguous images in the first image stream of the each training data set;
• training an image predictor using the first training data sets;
• generating second training data sets, each training data set including
• P contiguous images from a second image stream, where P is one of equal and not equal to N, the second image stream being either the same or different from the first image stream,
• an image that is one of(a) not an image immediately following the N contiguous images in the second image stream of the each training data set and (b) the image immediately following the P contiguous images in the second image stream of the each training data set;
• and
• a desired output that is a first value if(a) is true and a second value if(b) is true;
• and
• training an image discriminator according to the second training data sets;
• and
• training a control generator using outputs of the image discriminator.