Automatic photo editing via linguistic request
AssigneeAdobe Inc.
InventorsNing Xu, Zhe Lin, Franck Dernoncourt
Life of this patent
Estimating the term…
Abstract
The present disclosure relates to systems and methods for automatically processing images based on a user request. In some examples, a request is divided into a retouching command (e.g., a global edit) and an inpainting command (e.g., a local edit). A retouching mask and an inpainting mask are generated to indicate areas where the edits will be applied. A photo-request attention and a multi-modal modulation process are applied to features representing the image, and a modified image that incorporates the user's request is generated using the modified features.
Claims20
A method of image processing, comprising:
- identifying an image and an edit command for the image;
- encoding the edit command to obtain an inpainting vector and a retouching vector, wherein the inpainting vector indicates a local edit of the image and the retouching vector indicates a global edit of the image;
- generating an inpainting mask based on the inpainting vector and a retouching mask based on the retouching vector;
- generating an image feature representation based on the image and the inpainting mask;
- generating a modified image feature representation based on the image feature representation, the retouching mask, and an attention matrix calculated using the retouching vector;
- and
- generating a modified image based on the modified image feature representation, wherein the modified image represents an application of the edit command to the image.
A method of training a neural network for image processing, comprising:
- receiving training data comprising a training image, an edit command, and a ground truth image representing an application of the edit command to the training image;
- encoding the edit command to obtain an inpainting vector and a retouching vector wherein the inpainting vector indicates a local edit of the training image and the retouching vector indicates a global edit of the training image;
- generating an inpainting mask based on the inpainting vector and a retouching mask based on the retouching vector;
- generating an image feature representation based on the training image and the inpainting mask;
- generating a modified image feature representation based on the image feature representation, the retouching mask, and an attention matrix calculated using the retouching vector;
- and
- generating a modified image based on the modified image feature representation, wherein the modified image represents the application of the edit command to the training image;
- calculating a loss function based on the modified image and the ground truth image;
- and
- training the neural network based on the loss function.
An apparatus for image processing, comprising:
- an encoder configured to encode an edit command for an image to obtain an inpainting vector and a retouching vector, wherein the inpainting vector indicates a local edit of the image and the retouching vector indicates a global edit of the image;
- a phrase conditioned grounding (PCG) network configured to generate an inpainting mask based on the inpainting vector and a retouching mask based on the retouching vector;
- a convolutional network configured to generate an image feature representation based on the image and the inpainting mask;
- a multi-modal modulation network configured to generate a modified image feature representation based on the image feature representation, the retouching mask, and an attention matrix calculated using the retouching vector;
- and
- a decoder configured to generate a modified image based on the modified image feature representation, wherein the modified image represents an application of the edit command to the image.
Dependencies are inferred from the claim text.