Image manipulation using deep learning techniques in a patch matching operation

#This disclosure relates generally to computer-implemented methods and systems for computer graphics processing. Specifically, the present disclosure involves image manipulation using deep learning techniques in a patch matching operation.

#Image manipulation is the process of manipulating an image from one state to another. For example, image manipulation can include an image restoration process that is used to reproduce picture quality of an original image by removing or limiting noise present in a noisy image being restored. Existing image restoration techniques remove or limit noise in the noisy image using algorithms that require a hand-crafted design of a number of components (e.g., a patch distance metric that compares a small region of the noisy image with a reference image, a set of geometric transformations through which matches can be identified, post-processing voting operations to combine patches into a final restored image, and other design components). For example, a designer must manually configure parameters and variables of the restoration techniques.

#The hand-crafted design of these manipulation techniques is both technically challenging for the designer and brittle during implementation. For example, the hand-crafted design may rely on complex parameter tuning that is specific to a particular noisy image and a particular reference image. Additionally, the brittleness of the hand-crafted design may result in failures of the manipulation technique when any changes to the task associated with the hand-crafted design arise. As a result, even a slight change to the task for which the hand-crafted technique was designed can cause the image manipulation technique to fail.

#Features, embodiments, and advantages of the present disclosure are better understood when the following Detailed Description is read with reference to the accompanying drawings.

#FIG. 1 depicts an example of a computing environment for performing a patch matching operation with deep learning techniques to manipulate a target image, according to certain embodiments of the present disclosure.

#FIG. 2 depicts a simplified example of the patch matching operation of FIG. 1, according to certain embodiments of the present disclosure.

#The present disclosure involves image restoration using deep learning techniques in a patch matching operation. As discussed above, existing image restoration methods often generate unsatisfactory results especially when changes to a task performed by the hand-crafted image restoration occur. That is, the image restoration methods may fail when components of the task, such as a patch distance metric, a set of geometric transformations, post-processing, voting operations, or other design components, are changed. Certain embodiments described herein address these limitations by employing a deep learning framework through a patch matching operation. For instance, an image manipulation application uses trainable convolutional neural networks (CNNs) to extract features from a target image and a reference image received by the image manipulation application. Features identified by the trainable CNNs in the target image and the reference image may be matched using an exemplary patch matching algorithm described herein. That is, for each patch of the target image, a corresponding patch is identified in the reference image. These patches are combined and fed through one or more additional trainable CNNs for post-processing, and a final restored image is output from the image manipulation application. In some examples, other machine learning networks may be used in place of or in addition to the CNNs.

#Because the techniques described herein rely on deep learning algorithms, the techniques do not suffer from the brittle nature of the existing image restoration techniques. That is, the presently described techniques enables features of a patch matching operation to be learned in deep networks by making the patch matching operation differentiable. The differentiable patch matching operation is one that is able to rewrite itself based on information that the operation learns during training and use. Because the operation is rewritable during training and use, hand-crafting of the design of the patch matching operation and the brittle nature of the hand-crafted design are avoided.

#Other embodiments of the techniques described herein include image transformation. In an example, image transformation involves transforming an artwork of a particular style to a style of a photograph or transforming a photograph to an artwork of a particular style. In such an example, a reference image is used to assist in the transformation process. Additional embodiments include: image inpainting using a reference image as a guide; stylizing simulations, such as fluid simulations; stylizing a specific domain of an image, such as face images, to look like artwork (or vice versa); finding dense correspondences between two photos; or estimating the pose of an object in a photograph by using a reference object as auxiliary information. In an example, the image may have also undergone geometric transformation, such as changes to the scale or rotation of the image, or a camera viewpoint of an image may be different.