Real scene image editing method based on hierarchically classified text guidance

A real scene image editing method based on hierarchically classified text guidance, comprising:

• step 1:
• selecting a hierarchical multi-label text classification model, inputting a primary word t1 to the hierarchical multi-label text classification model to hierarchically classify an indoor style description, wherein an output of the hierarchical multi-label text classification model is set with three levels:
• the primary word t1 is an abstract style description;
• a secondary word t2 is a composition description of a scene image;
• and a tertiary word t3 is a detailed description corresponding to an abstract style;
• the composition description comprises a layout, an abstract, an attribute, and a color;
• and
• the detailed description comprises specific descriptions of the layout, the abstract, the attribute, and the color;
• step 2:
• utilizing an e4e inversion model to obtain a latent vector w of an indoor image trained in a large-scale scene understanding (LSUN) dataset, w∈W+, W+ representing a vector space;
• and segmenting the latent vector w based on a semantic hierarchical characteristic of StyleGAN and in combination with the secondary word t2 obtained in step 1;
• wherein step 2 specifically comprises:
• 2-1, utilizing an e4e model trained on the LSUN dataset to obtain an inverse latent vector w of a real indoor scene in a format of .pt file as an input to the StyleGAN;
• and
• 2-2, dividing the obtained latent vector w according to the semantic hierarchical characteristic of the StyleGAN, wherein the layout corresponds to [0,2) layer of a generative network;
• the object corresponds to [2.6) layer of the generative network;
• the attribute corresponds to [6,12) layer of the generative network;
• and the color corresponds to [12,14) layer of the generative network;
• and
• step 3:
• training a plurality of latent space residual mappers, wherein since different StyleGAN layers are known to generate details of different levels in the scene image, the plurality of latent space residual mappers are divided into four groups, each group comprising a single latent space residual mapper, and the four groups are configured to correspondingly generate details of the layout, the abstract, the attribute, and the color;
• and realizing manipulation on a real scene image with a visual abstract text by utilizing the tertiary word t3 obtained in step 1 and a contrastive language-image pre-training (CLIP) model.