Semantically-consistent image style transfer

A method of training source domain low-level encoder neural network layers, high-level encoder neural network layers, high-level decoder neural network layers, target domain low-level decoder neural network layers, target domain low-level encoder neural network layers and source domain low-level encoder neural network layers, the method comprising:

• receiving a training input source domain image from a source domain;
• processing the training input source domain image using the source domain low-level encoder neural network layers to generate a low-level representation of the training input source domain image;
• processing the low-level representation using the high-level encoder neural network layers to generate an embedding of the training input source domain image;
• processing the embedding of the training input source domain image using the one or more high-level decoder neural network layers to generate a high-level feature representation of features of the training input source domain image;
• and
• processing the high-level feature representation of the features of the training input source domain image using the one or more target domain low-level decoder neural network layers to generate a training output target domain image that is from a target domain;
• processing the training output target domain image using the one or more target domain low-level encoder neural network layers to generate a low-level representation of the training output target domain image;
• processing the low-level representation using the one more high-level encoder neural network layers to generate an embedding of the training output target domain image;
• determining a first gradient of a semantic consistency loss function that reduces a distance measure between the embedding of the training output target domain image and the embedding of the training input source domain image;
• and
• updating, using the first gradient, current values of the parameters of the source domain low-level encoder neural network layers, the high-level encoder neural network layers, the high-level decoder neural network layers, the target domain low-level decoder neural network layers, and the target domain low-level encoder neural network layers.

A system comprising one or more computers and one or more storage devices storing instructions that when implemented by the one or more computers cause the one or more computers to perform operations comprising:

• receiving an input source domain image from a source domain;
• processing the source domain image using one or more source domain low-level encoder neural network layers that are specific to images from the source domain to generate a low-level representation of the input source domain image;
• processing the low-level representation using one more high-level encoder neural network layers that are shared between images from the source and target domains to generate an embedding of the input source domain image;
• processing the embedding of the input source domain image using one or more high-level decoder neural network layers that are shared between images from the source and target domains to generate a high-level feature representation of features of the input source domain image;
• and
• processing the high-level feature representation of the features of the input source domain image using one or more target domain low-level decoder neural network layers that are specific to generating images from the target domain to generate an output target domain image that is from the target domain but that has similar semantics to the input source domain image.