# US 12,002,136 B2 · Style try-on with reverse GANs > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,002,136 B2 - Document ID: US12002136B2 - Status: Granted - Kind: B2 - Application number: 17/685,675 - Filed: 2022-03-03 - Published: 2022-09-08 - Granted: 2024-06-04 - Earliest claimed: 2021-03-03 - Assignees: L'oreal - Applicants: L'oreal - Inventors: Rohit Saha; Brendan Duke - Web page: https://patentgrep.com/patent/US12002136B2 ## CPC - G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0475: Generative networks - G06N 3/08: Learning methods - G06N 3/094: Adversarial learning - G06Q 30/0631: {Recommending goods or services} - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 7/00: Image analysis - G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215) - G06T 7/11: Region-based segmentation - G06T 11/60: Creating or editing images; Combining images with text - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06T 2200/24: involving graphical user interfaces [GUIs] - G06T 2207/20076: Probabilistic image processing - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20212: Image combination - G06T 2207/30196: Human being; Person - G06T 2207/30201: Face - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Hairstyle transfer is challenging due to hair structure differences in the source and target hair. Latent Optimization of Hairstyles via Orthogonalization (LOHO) is an optimization-based approach using GAN inversion to infill missing hair structure details in latent space during hairstyle transfer. Hair is decomposed into three attributes: perceptual structure, appearance, and finer style, and includes tailored losses to model each of these attributes independently. Two-stage optimization and gradient orthogonalization enable disentangled latent space optimization of the three hair attributes. Using LOHO for latent space manipulation, users can synthesize novel photorealistic images by manipulating hair attributes either individually or jointly, transferring the desired attributes from reference hairstyles. ## Claims ### Claim 1 A method to perform a style transfer of a style using artificial intelligence (AI), the style comprising a plurality of style attributes, the method comprising: processing a plurality of images comprising a first image (I1), a second image (I2) and a third image (I3) using an AI network framework comprising a generative adversarial network (GAN) generator and a two stage optimization to generate a synthesized image (IG) comprising an identity represented by the first image (I1) and a style determined from at least one style attribute represented by the second image (I2) and at least one style attribute represented by the third image (I3); wherein the AI network framework is configured to optimize a latent space of a GAN model to perform the style transfer while disentangling the at least one style attribute represented by the second image (I2) and the at least one style attribute represented by the third image (I3); and wherein the network framework performs gradient orthogonalization in the two stage optimization to disentangle the at least one style attribute represented by second image (I2) and the at least one style attribute represented by the third image (I3). ### Claim 2 (depends on 1) The method of claim 1, wherein IG comprises an identity region, a style region and a background region and wherein, in a first stage according to an objective function to optimize the latent space, the network framework is configured to reconstruct the identity represented by I1 to the identity region of IG; and reconstruct the at least one style attribute represented by I2 to the style region of IG. ### Claim 3 (depends on 2) The method of claim 2, wherein, in a second stage according to the objective function, the AI network framework is configured to: transfer respectively each of the at least one style attribute represented by I3 to the style region of IG. ### Claim 4 (depends on 1) The method of claim 1, wherein: the style is a hairstyle; the at least one style attribute represented by I2 is a hair shape and structure attribute; the at least one style attribute represented by I3 is i) an appearance attribute and ii) a finer style attribute. ### Claim 5 (depends on 1) The method of claim 1, wherein the network framework is configured to provide editable hairstyle transfer, disentangling features of a) hair shape and structure; and b) hair appearance and hair finer style thereby to permit a choice of hair attribute to transfer. ### Claim 6 (depends on 5) The method of claim 5, wherein any of: the identity of I1 is unique between I1, I2 and I3, thereby to perform a complete hairstyle transfer; the hair shape and structure of I2 is unique between I1, I2 and I3, thereby to perform at least a hairstyle transfer related to shape and structure; the hair appearance of I3 is unique between I1, I2 and I3 thereby to perform at least a hairstyle transfer related to appearance; and the hair finer style of I3 is unique between I1, I2 and I3 thereby to perform at least a hairstyle transfer related to finer details of hair. ### Claim 7 (depends on 1) The method of claim 1 comprising: processing IG, I1, I2 and I3 respectively using a segmentation network to define respective hair masks and face masks for each image and using selected one of such masks to define respective target masks for transferring the style. ### Claim 8 (depends on 1) The method of claim 1, wherein: the GAN generator initially generates IG as a mean image to receive the style transfer; the identity is reconstructed using high-level features extracted by processing I1 using a pretrained neural network encoder; and, in a hairstyle transfer, hair shape and structure are reconstructed using features from later blocks generated by processing I2 using the pretrained neural network encoder. ### Claim 9 (depends on 1) The method of claim 1, wherein in a hairstyle transfer, hair appearance comprises colour, and hair finer style comprises finer details comprising any of wisp styles and shading variations between hair strands. ### Claim 10 (depends on 1) The method of claim 1, wherein the method comprises any of providing an interface to an e-commerce service with which to purchase a hair or hairstyle product, a hair or hairstyle service or both such product and service; and providing an interface to recommend such product, service or both. ### Claim 11 (depends on 1) The method of claim 1, wherein the IG is provided for display in a graphical user interface for comparison with I1. ### Claim 12 (depends on 1) The method of claim 1, comprising: providing an interface to receive I1; providing a store of reference images showing respective style attributes, including a hairstyle, comprising hair shape and structure, and hair appearance and finer style; providing a selection interface to receive input to define I2 from one of the reference images; and providing a selection interface to receive input to define I3 from one of the reference images. ### Claim 13 A computing device comprising a processor and a storage device storing computer-readable instructions that, when executed by the processor, cause the computing device to: transfer a hairstyle via an artificial intelligence (AI) network framework comprising a generative adversarial network (GAN) generator that generates a synthesized image (IG) comprising transferred hair attributes from reference images to an identity from a face of a first image (I1), the hair attributes comprising i) hair shape and structure, ii) a hair appearance and iii) a hair finer style; wherein the network framework is configured to optimize a latent space to disentangle the hair attributes i) hair shape and structure from ii) hair appearance and iii) hair finer style; and wherein the GAN generator is trained using two-stage optimization and gradient orthogonalization to enable the latent space optimization to disentangle the hair attributes. ### Claim 14 (depends on 13) The computing device of claim 13, wherein the reference images comprise a second image (I2) and a third image (I3); each of I1, I2 and I3 comprise portrait images; and the network framework uses the hair shape and structure extracted from I2; and each of hair appearance and hair finer style respectively extracted from I3. ### Claim 15 (depends on 13) The computing device of claim 13, wherein the instructions when executed cause the computing device to provide one or both of: an interface to an e-commerce service with which to purchase a product and/or service associated with a hairstyle; and an interface to recommend a hair or hairstyle product, a hair or hairstyle service or both of such product and service. ### Claim 16 (depends on 13) The computing device of claim 13, wherein the instructions when executed cause the computing device to: provide an interface to receive I1; provide a store of reference images showing respective hair attributes; provide a selection interface to receive input to select at least one reference image to define the hair attributes for the hairstyle transfer. ### Claim 17 A computing device comprising circuitry that, when operated, causes the computing device to: provide a network framework to perform a virtual hairstyle try-on relative to an identity image and a plurality of reference images representing different hairstyle attributes to simulate a hairstyle on an identity, the network framework configured to perform an optimization that disentangles the different hairstyle attributes to provide realistic synthesized hair when incorporating the identity and hairstyle into a synthesized image representing the virtual hairstyle try-on; and provide the synthesized image for presentation; wherein the network framework is trained using two-stage optimization and gradient orthogonalization to enable the latent space optimization to disentangle the hair attributes. ### Claim 18 (depends on 17) The computing device of claim 17, wherein the circuitry, when operated, causes the computing device to at least one of: provide an interface to purchase a hair or hairstyle product, hair or hairstyle service or both of such product and service; and provide an interface to generate hair or hairstyle recommendations.