Exemplar-based object appearance transfer driven by correspondence

A method for training a machine learning model, comprising:

• receiving a training set including a content image and a style image;
• encoding the content image and the style image using a machine learning model to obtain content features and style features;
• computing a pixel-wise similarity score between a pixel from the content image and a pixel from the style image based on the content features and the style features;
• including the pixel from the content image in a sparse set of pixels of the content image based on the pixel-wise similarity score;
• aligning the content features and the style features to obtain a sparse correspondence map that indicates a correspondence between the sparse set of pixels of the content image and corresponding pixels of the style image;
• generating a hybrid image based on the sparse correspondence map, wherein the hybrid image depicts an object from the content image and an appearance attribute from the style image;
• and
• updating parameters of the machine learning model based on the hybrid image.

An apparatus for image processing, comprising:

• an image encoder configured to encode a content image and a style image using a machine learning model to obtain content features and style features, wherein the content image includes a first object having a first appearance attribute and the style image includes a second object having a second appearance attribute;
• an alignment component configured to align the content features and the style features to obtain a sparse correspondence map that indicates a correspondence between a sparse set of pixels of the content image and corresponding pixels of the style image;
• and
• an image decoder configured to generate a hybrid image based on the sparse correspondence map by identifying a layer index for the image encoder, providing the content features as a first input to at least one layer of the image decoder having a first index that is less than the layer index, combining the sparse correspondence map with the style features to obtain aligned style features, and providing the aligned style features as a second input to at least one subsequent layer of the image decoder having a second index that is greater than the layer index, wherein the hybrid image depicts the first object having the second appearance attribute.