Image generation with multiple image editing modes

An apparatus for image processing, comprising:

• a processor and a memory storing instructions and in electronic communication with the processor, the processor being configured to execute the instructions to:
• obtain an input image and a mask indicating an edit region of the image;
• identify a mask type of the mask from a plurality of predetermined mask types corresponding to a plurality of image editing modes;
• identify an image editing mode from the plurality of image editing modes of an image generation network based on the mask type by setting a configuration of the image generation network, wherein the plurality of image editing modes includes at least one inpainting mode and at least one outpainting mode;
• identify a plurality of conditions corresponding to the image editing mode;
• compute a conditional embedding such that the mask is conditionally applied to certain data points or features of the input image that meet the plurality of conditions;
• generate a partial image based on the mask;
• and
• generate an output image using the image generation network based on the input image, the conditional embedding, and the mask, by combining the input image and the partial image and performing an image editing task corresponding to the image editing mode according to the configuration of the image generation network, wherein the image generation network is trained to perform a plurality of image editing tasks corresponding to the plurality of image editing modes.

A method for image processing, comprising:

• identifying a mask type from a plurality of predetermined mask types corresponding to a plurality of image editing modes;
• obtaining training data including a plurality of training samples corresponding to the plurality of image editing modes, respectively, based on the mask type by setting a configuration of an image generation network, wherein the plurality of image editing modes includes at least one inpainting mode and at least one outpainting mode;
• identifying a plurality of conditions corresponding to each of the plurality of image editing modes;
• generating a conditional embedding for each of the training samples based on the plurality of conditions, such that a mask conditionally applies to certain data points or features of an input image that meet the plurality of conditions;
• and
• training the image generation network to generate a partial image based on the mask, and generate output images based on the conditional embedding and the mask type by combining an input image and the partial image, and performing an image editing task corresponding to each of the plurality of image editing modes according to the configuration of the image generation network using the training data.