# US 2021/0241531 A1 · Method and apparatus for providing virtual clothing wearing service based on deep-learning > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0241531 A1 - Document ID: US20210241531A1 - Status: Published application - Kind: A1 - Application number: 17/105,599 - Filed: 2020-11-26 - Published: 2021-08-05 - Earliest claimed: 2020-02-04 - Assignees: Nhn Corporation - Applicants: Nhn Corporation - Inventors: Hyug Jae Lee; Rok Kyu Lee; Min Seok Kang; Myoung Hoon Cho - Web page: https://patentgrep.com/patent/US20210241531A1 ## CPC - G06T 3/08: Projecting images onto non-planar surfaces, e.g. geodetic screens - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0475: Generative networks - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 3/094: Adversarial learning - G06N 20/00: Machine learning - G06Q 30/0643: {graphically representing goods, e.g. 3D product representation} - G06T 3/18: Image warping, e.g. rearranging pixels individually - G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics - G06V 10/764: using classification, e.g. of video objects - G06V 10/776: Validation; Performance evaluation - G06V 10/82: using neural networks - G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method and apparatus provide a virtual clothing wearing service based on deep-learning. A virtual clothing wearing server based on deep-learning includes a communicator configured to receive a user image and a v clothing image; a memory configured to store a program including first and second deep-learning models; a processor configured to generate an image of virtually dressing a virtual wearing clothing on a user. The program is configured to: generate, by the first deep-learning model, a transformed virtual wearing clothing image by transforming the virtual wearing clothing image in accordance with a body of the user in the user image based on the user image and the virtual wearing clothing image, and generate, by the second deep-learning model, the virtual wearing person image by dressing the transformed virtual wearing clothing on the body of the user based on the user image and the transformed virtual wearing clothing image. ## Claims ### Claim 1 A server for generating an virtual clothing wearing image based on deep-learning, comprising: a communicator configured to receive a user image and a clothing image for virtual wearing; a memory configured to store a virtual clothing wearing deep-learning model including a first deep-learning model and a second deep-learning model; a processor configured to generate a virtual wearing image of virtually dressing a cloth, included the clothing image, on a user, included in the user image, using the virtual clothing wearing deep-learning model, wherein the processor is configured to: generate, by the first deep-learning model, an image of a transformed virtual wearing clothing by transforming the received clothing image in accordance with a body of the user included in the received user image, and generate, by the second deep-learning model, the virtual wearing image by dressing the transformed virtual wearing clothing, generated by the first deep-learning model, on the body of the user of the received user image. ### Claim 2 (depends on 1) The virtual clothing wearing server based on the deep-learning of claim 1, wherein the first deep-learning model includes a first-1 deep-learning model and a first-2 deep-learning model, wherein the first-1 deep-learning model is configured to generate a first-1 transformation virtual wearing clothing image by performing Perspective Transformation to the received clothing image to match with a direction of the body of the user included in the received user image, and wherein the first-2 deep-learning model is configured to generate an image of first-2 transformation virtual wearing clothing by transforming the first-1 transformation virtual wearing clothing, included in the image of the first-1 transformation virtual wearing clothing, to be matched with a shape of the body of the user included in the received user image. ### Claim 3 (depends on 2) The virtual clothing wearing server based on the deep-learning of claim 2, wherein the second deep-learning model includes a second-1 deep-learning model configured to generate a synthesis mask image and an intermediate person image based on the image of first-2 transformation virtual wearing clothing and the received user image, and wherein the second deep-learning model is configured to generate a first virtual wearing person image by synthesizing the synthesis mask image, the intermediate person image and the image of first-2 transformation virtual wearing clothing. ### Claim 4 (depends on 3) The virtual clothing wearing server based on the deep-learning of claim 3, wherein the synthesis mask image is an image that a position of the first-2 transformation virtual wearing clothing is territorialized on the received user image, and wherein the intermediate person image is an image of an arm and/or a hand of the body of the user which is generated based on a length of the first-2 transformation virtual wearing clothing. ### Claim 5 (depends on 4) The virtual clothing wearing server based on the deep-learning of claim 4, wherein the second deep-learning model further includes a second-2 deep-learning model configured to generate a second virtual wearing person image which is generated by a plurality of dilated convolutions and based on the user image and the first virtual wearing person image. ### Claim 6 (depends on 5) The virtual clothing wearing server based on the deep-learning of claim 5, wherein the communicator is configured to receive a training data set including a person, a cloth for virtual wearing, a transformation clothing truth image when wearing, and a Truth Label for the person dressing the cloth for virtual wearing. ### Claim 7 (depends on 6) The virtual clothing wearing server based on the deep-learning of claim 6, wherein the processor is configured to: perform training of the first-1 deep-learning model using a first-1 model loss of comparing the first-1 transformation virtual wearing clothing image generated by the first-1 deep-learning model and the transformation clothing truth image when virtually wearing for the person and the cloth for virtual wearing of the training data set, and perform training of the first deep-learning model using a first-2 model loss of comparing the image of the first-2 transformation virtual wearing clothing generated by the first-2 deep-learning model and the transformation clothing truth image when virtually wearing for the person and the cloth for virtual wearing of the training data set, when training the first-1 deep-learning model. ### Claim 8 (depends on 6) The virtual clothing wearing server based on the deep-learning of claim 6, wherein the processor is configured to perform training of the first deep-learning model using a grid interval consistency loss based on a distance between pixels of an image of the cloth for virtual wearing. ### Claim 9 (depends on 8) The virtual clothing wearing server based on the deep-learning of claim 8, wherein the processor is configured to generate an occlusion clothing image on which an occlusion part is removed from the transformation clothing truth image through an occlusion process, and wherein the first deep-learning model is configured to use the occlusion clothing image when training using the first-2 model loss. ### Claim 10 (depends on 8) The virtual clothing wearing server based on the deep-learning of claim 8, wherein the processor is configured to: generate a first discrimination image based on the image of the first-2 transformation virtual wearing clothing, and perform training of the first deep-learning model using a first adversarial loss based on the first discrimination image. ### Claim 11 (depends on 10) The virtual clothing wearing server based on the deep-learning of claim 10, wherein the processor is configured to generate a second virtual wearing person image by the second-2 deep-learning model using the image of first-2 transformation virtual wearing clothing generated by the first-2 deep-learning model for the person and the cloth for virtual wearing of the training data set. ### Claim 12 (depends on 11) The virtual clothing wearing server based on the deep-learning of claim 11, wherein the processor is configured to perform training of the second deep-learning model using the second model loss by comparing the second virtual wearing person image for the person and the cloth for virtual wearing of the training data set and the Truth Label for the person dressing the cloth for virtual wearing. ### Claim 13 (depends on 12) The virtual clothing wearing server based on the deep-learning of claim 12, wherein the memory further includes VGG-19 neural network, and wherein the processor is configured to generate a layer property map for the second virtual wearing person image for the person and the cloth for virtual wearing of the training data set and a layer property map for the Truth Label for the person dressing the cloth for virtual wearing. ### Claim 14 (depends on 13) The virtual clothing wearing server based on the deep-learning of claim 13, wherein the processor is configured to perform training of the second deep-learning model using a perceptual loss by comparing the layer property map for the second virtual wearing person image for the person and the cloth for virtual wearing of the training data set and the layer property map for the Truth Label for the person dressing the cloth for virtual wearing. ### Claim 15 (depends on 14) The virtual clothing wearing server based on the deep-learning of claim 14, wherein the processor is configured to: generate a second discrimination image based on the second virtual wearing person image through the second deep-learning model, and perform training using a second adversarial loss based on the second discrimination image. ### Claim 16 A terminal, comprising: a communicator configured to transmit a user image and a clothing image for virtual wearing; at least one processor configured to provide a virtual wearing image of virtually dressing a cloth, included in the clothing image, to a user, included in the user image, using a virtual clothing wearing service request program; and a memory configured to store the virtual clothing wearing service request program that, if executed by the at least one processor, configure the at least one processor to: select the user image and the clothing image for virtual wearing, transmit the selected user image and the selected clothing image using the communicator, and receive a virtual wearing person image generated by a virtual clothing wearing server based on deep-learning through the communicator, wherein the virtual clothing wearing server based on the deep-learning includes a first deep-learning model and a second deep-learning model, the first deep-learning model is configured to generate an image of a transformed virtual wearing clothing by transforming the clothing image in accordance with a body of the user included in the user image, and the second deep-learning model is configured to generate the virtual wearing person image by dressing the transformed virtual wearing clothing, generated by the first deep-learning model, on the body of the user included in the user image. ### Claim 17 A method for providing a virtual clothing wearing service by a virtual clothing wearing server based on deep-learning, the method comprising: obtaining a user image and a clothing image for virtual wearing; inputting the user image and the clothing image to a first deep-learning model; outputting, by the first deep-learning model, an image of a transformed virtual wearing clothing which is transformed in accordance with a body of the user included in the user image by the first deep-learning model; inputting the user image and the image of the transformed virtual wearing clothing to the second deep-learning model; and outputting, by the second deep-learning model, a virtual wearing person image by dressing the transformed virtual wearing clothing, outputted by the first deep-learning model, on the body of the user included in the user image. ### Claim 18 (depends on 17) The method for providing the virtual clothing wearing service by the virtual clothing wearing server based on the deep-learning of claim 17, wherein the outputting, by the first deep-learning model, of the image of the transformed virtual wearing clothing includes: generating, by a first-1 deep-learning model, a first-1 transformation virtual wearing clothing image by performing Perspective Transformation to the clothing image to match with a direction of the body of the user included in the user image. ### Claim 19 (depends on 18) The method for providing the virtual clothing wearing service by the virtual clothing wearing server based on the deep-learning of claim 18, wherein the outputting, by the first deep-learning model, of the image of the transformed virtual wearing clothing further includes: generating, by a first-2 deep-learning model, a first-2 transformation virtual wearing clothing image by transforming the first-1 transformation virtual wearing clothing, included in the image of the first-1 transformation virtual wearing clothing, to be matched with a shape of the body of the user, included in the obtained user image. ### Claim 20 (depends on 17) The method for providing the virtual clothing wearing service by the virtual clothing wearing server based on the deep-learning of claim 17, further comprising transmitting the virtual wearing image, outputted by the second deep-learning model, to a terminal of the user.