# US 2022/0044451 A1 · Method and device for image generation and colorization > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0044451 A1 - Document ID: US20220044451A1 - Status: Published application - Kind: A1 - Application number: 17/122,680 - Filed: 2020-12-15 - Published: 2022-02-10 - Earliest claimed: 2020-08-04 - Assignees: Ping An Technology (Shenzhen) Co., Ltd. - Applicants: Ping An Technology (Shenzhen) Co., Ltd. - Inventors: Yuchuan GOU; Minghao LI; Bo GONG; Mei HAN - Web page: https://patentgrep.com/patent/US20220044451A1 ## CPC - G06T 11/00: Two-dimensional [2D] image generation - G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0475: Generative networks - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06N 3/09: Supervised learning - G06N 3/094: Adversarial learning - G06V 10/56: relating to colour - G06V 10/7788: {the supervisor being a human, e.g. interactive learning with a human teacher} - G06V 10/82: using neural networks - G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations - G06V 30/274: {Syntactic or semantic context, e.g. balancing} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method and device for image generation and colorization are provided. The method includes: displaying a drawing board interface; obtaining semantic labels of an image to be generated based on user input on the drawing board interface, each semantic label indicating a content of a region in the image to be generated; obtaining a color feature of the image to be generated; and automatically generating the image using a generative adversarial network (GAN) model according to the semantic labels and the color feature. The color feature is a latent vector input to the GAN model. ## Claims ### Claim 1 A method for image generation and colorization, applied to a computing device, comprising: displaying a drawing board interface; obtaining semantic labels of an image to be generated based on user input on the drawing board interface, each semantic label indicating a content of a region in the image to be generated; obtaining a color feature of the image to be generated; and automatically generating the image using a generative adversarial network (GAN) model according to the semantic labels and the color feature, the color feature being a latent vector input to the GAN model. ### Claim 2 (depends on 1) The method according to claim 1, wherein obtaining the color feature comprises: obtaining a stroke image having a same size as the image to be generated, the stroke image including one or more colored strokes input by a user; and extracting the color feature based on the stroke image. ### Claim 3 (depends on 2) The method according to claim 2, wherein when no user input is obtained for the stroke image, the color feature is extracted from a default stroke image having a uniform color. ### Claim 4 (depends on 2) The method according to claim 2, wherein obtaining the stroke image comprises: displaying candidate colors on the drawing board interface; obtaining a target color based on a user selection among the candidate colors; detecting a dragging operation; and recording a colored stroke of the stroke image according to the target color and positions corresponding to the dragging operation. ### Claim 5 (depends on 1) The method according to claim 1, wherein obtaining the semantic labels comprises: obtaining a semantic map having a same size as the image to be generated, the semantic map including a plurality of regions, pixels in a same region having a same semantic label indicating the content of the region. ### Claim 6 (depends on 5) The method according to claim 5, wherein obtaining the semantic labels further comprises: producing the semantic map based on user drawing operations on the drawing board interface; receiving semantic label texts assigned by a user to the plurality of regions of the semantic map; and obtaining the semantic labels according to the user-drawn semantic map and the assigned semantic label texts. ### Claim 7 (depends on 5) The method according to claim 5, wherein obtaining the semantic labels further comprises: modifying the semantic map according to a user operation on the drawing board interface, including at least one of: modifying a first semantic label text corresponding to one or more pixels of the semantic map, or modifying a location of a region corresponding to a second semantic label text. ### Claim 8 (depends on 5) The method according to claim 5, wherein obtaining the semantic labels further comprises: displaying multiple semantic map templates on the drawing board interface; and obtaining the semantic labels according to a user selection on one of the multiple semantic map templates. ### Claim 9 (depends on 1) The method according to claim 1, further comprising: displaying the automatically-generated image on the drawing board interface, the automatically-generated image being a first image; obtaining a modification instruction on at least one of the semantic labels or the color feature; and generating a modified image using the GAN model according to the semantic labels and the color feature updated based on the modification instruction. ### Claim 10 (depends on 9) The method according to claim 9, further comprising: displaying the modified image on the drawing board interface; receiving a return instruction; and displaying the first image on the drawing board interface in response to the return instruction. ### Claim 11 A device for image generation and colorization, comprising: a memory; and a processor coupled to the memory and configured to perform: displaying a drawing board interface; obtaining semantic labels of an image to be generated based on user input on the drawing board interface, each semantic label indicating a content of a region in the image to be generated; obtaining a color feature of the image to be generated; and automatically generating the image using a generative adversarial network (GAN) model according to the semantic labels and the color feature, the color feature being a latent vector input to the GAN model. ### Claim 12 (depends on 11) The device according to claim 11, wherein obtaining the color feature comprises: obtaining a stroke image having a same size as the image to be generated, the stroke image including one or more colored strokes input by a user; and extracting the color feature based on the stroke image. ### Claim 13 (depends on 12) The device according to claim 12, wherein when no user input is obtained for the stroke image, the color feature is extracted from a default stroke image having a uniform color. ### Claim 14 (depends on 11) The device according to claim 11, wherein obtaining the semantic labels comprises: obtaining a semantic map having a same size as the image to be generated, the semantic map including a plurality of regions, pixels in a same region having a same semantic label indicating the content of the region. ### Claim 15 (depends on 14) The device according to claim 14, wherein obtaining the semantic labels further comprises: producing the semantic map based on user drawing operations on the drawing board interface; receiving semantic label texts assigned by a user to the plurality of regions of the semantic map; and obtaining the semantic labels according to the user-drawn semantic map and the assigned semantic label texts. ### Claim 16 (depends on 14) The device according to claim 14, wherein obtaining the semantic labels further comprises: modifying the semantic map according to a user operation on the drawing board interface, including at least one of: modifying a first semantic label text corresponding to one or more pixels of the semantic map, or modifying a location of a region corresponding to a second semantic label text. ### Claim 17 (depends on 14) The device according to claim 14, wherein obtaining the semantic labels further comprises: displaying multiple semantic map templates on the drawing board interface; and obtaining the semantic labels according to a user selection on one of the multiple semantic map templates. ### Claim 18 (depends on 11) The device according to claim 11, wherein the processor is further configured to perform: displaying the automatically-generated image on the drawing board interface, the automatically-generated image being a first image; obtaining a modification instruction on at least one of the semantic labels or the color feature; and generating a modified image using the GAN model according to the semantic labels and the color feature updated based on the modification instruction. ### Claim 19 (depends on 18) The device according to claim 18, wherein the processor is further configured to perform: displaying the modified image on the drawing board interface; receiving a return instruction; and displaying the first image on the drawing board interface in response to the return instruction. ### Claim 20 A non-transitory computer readable storage medium, storing computer instructions that, when being executed by a processor, causing the processor to perform: displaying a drawing board interface; obtaining semantic labels of an image to be generated based on user input on the drawing board interface, each semantic label indicating a content of a region in the image to be generated; obtaining a color feature of the image to be generated; and automatically generating the image using a generative adversarial network (GAN) model according to the semantic labels and the color feature, the color feature being a latent vector input to the GAN model.