# US 2021/0241500 A1 · Method and system for prov iding photorealistic changes for digital image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241500 A1
- Document ID: US20210241500A1
- Status: Published application
- Kind: A1
- Application number: 17/050,587
- Filed: 2019-04-23
- Published: 2021-08-05
- Earliest claimed: 2018-04-24
- Assignees: Metail Limited
- Applicants: Metail Limited
- Inventors: Yu CHEN; Jim DOWNING; Tom ADEYOOLA; Sukrit SHANKER
- Web page: https://patentgrep.com/patent/US20210241500A1

## CPC

- G06T 11/00: Two-dimensional [2D] image generation
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 20/00: Machine learning
- G06Q 30/0643: {graphically representing goods, e.g. 3D product representation}
- G06T 3/18: Image warping, e.g. rearranging pixels individually
- G06T 7/11: Region-based segmentation
- G06T 7/194: involving foreground-background segmentation
- G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77)
- G06T 2207/20081: Training; Learning
- G06T 2210/16: Cloth
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Disclosed is a computer-implemented method for providing photorealistic changes for a digital image. The method includes receiving a digital image of dressable model, receiving digital cutout garment textures that are indexed according to an outfitting layering order and aligned with body shape and pose of the dressable model, receiving binary silhouettes of the digital cutout garment textures, generating a garment layer index mask by compositing the binary silhouettes of the digital cutout garment textures indexed according to the outfitting layering order, receiving a composite image obtained by overlaying the digital cutout garment textures according to the indexed outfitting layering order on the digital image of the dressable model, inputting the composite image and the garment layer index mask into a machine learning system for providing photorealistic changes, and receiving from the machine learning system a digital file including photorealistic changes for application to the composite image.

## Claims

### Claim 1

A computer-implemented method for providing photorealistic changes for a digital image, the method including steps of:
(i) receiving a digital image of a dressable model;
(ii) receiving one or more digital cutout garment textures, indexed according to an outfitting layering order, aligned with a body shape and a pose of the dressable model;
(iii) receiving binary silhouettes of the one or more digital cutout garment textures;
(iv) generating a garment layer index mask by compositing the binary silhouettes of the one or more digital cutout garment textures, indexed according to the outfitting layering order;
(v) receiving a composite image obtained by overlaying the one or more digital cutout garment textures, according to the indexed outfitting layering order, on the digital image of the dressable model;
(vi) inputting the composite image and the garment layer index mask into a machine learning system for providing photorealistic changes; and
(vii) receiving from the machine learning system a digital file including photorealistic changes, suitable for application to the composite image;
the method including the step of storing the digital file including the photorealistic changes.

### Claim 2

(canceled)

### Claim 3 (depends on 1)

The method of claim 1, wherein the step (i) includes generating or receiving a binary mask of the digital image of the dressable model, and wherein step (vi) includes inputting the binary mask of the digital image of the dressable model into the machine learning system for providing photorealistic changes.

### Claim 4 (depends on 1)

The method of claim 1, including a step in which the received composite image is generated by overlaying the one or more digital cutout garment textures, according to the indexed outfitting layering order, on the digital image of the dressable model.

### Claim 5 (depends on 1)

The method of claim 1, including steps in which
(a) each of the one or more digital cutout garment textures is generated by separating or cutting out a garment in a foreground of the image from a garment image background, and then
(b) is transformed or warped to match or align with the body shape and pose of the model.

### Claim 6 (depends on 1)

The method of claim 1, including a step of generating a quality-enhanced composed image by applying the digital file including photorealistic changes, to the composite image.

### Claim 7

(canceled)

### Claim 8 (depends on 1)

The method of claim 1, in which the photorealistic changes are one or more of: a shadow or shadows, a crease or creases, a scene background or scene backgrounds, a scene setting or scene settings, an animation or animations, or lighting adaption.

### Claim 9 (depends on 1)

The method of claim 1, including a step in which the machine learning system is trained.

### Claim 10 (depends on 9)

The method of claim 9, (a) including a step in which to train the machine learning system at least 1,000 images are used; or
(b) in which the step in which the machine learning system is trained includes a multi-stage iterative training process wherein the machine learning system is initially trained using a model for low-resolution-images, and then is gradually re-trained or fine tuned for greater image resolutions through weight-sharing to produce a training model.

### Claim 11

(canceled)

### Claim 12 (depends on 1)

The method of claim 1, wherein the digital images are 2D images.

### Claim 13 (depends on 1)

The method of claim 1, (a) wherein the method generates ground truth diff shadow image by subtracting a final composed image from an intermediate composed image obtained after overlaying one or more digital cutout garment textures, according to indexed outfitting layering order, on digital image of dressable model, for training the machine learning system; or
(b) wherein the digital image of a dressable model is selected based on a predetermined selection criterion or criteria; or
(c) wherein each binary silhouette is generated by extracting and thresholding alpha channel of a respective digital cutout garment texture; or
(d) in which background regions and regions containing the undressed model or any garments or accessories within the undressed model image are all indexed with the lowest layering order index value.

### Claim at position 14

14-16. (canceled)

### Claim 17 (depends on 1)

The method of claim 1, wherein the garment layer index mask conveys important spatial information that guides the machine learning system on how and where to precisely render photo realistic features.

### Claim 18 (depends on 17)

The method of claim 17, wherein the garment layer index mask conveys important spatial information that guides the machine learning system on, in which direction such photo realistic features should be added.

### Claim 19 (depends on 1)

The method of claim 1, wherein
subsequent image training inputs are used to improve prediction quality of the machine learning system.

### Claim 20 (depends on 19)

The method of claim 19, (a) wherein the subsequent image training inputs include one or more of: an undressed model image; an undressed model image in format of a three channel RGB image; a body mask including a binary foreground and background segmentation of the undressed model image; a body mask including a binary foreground and background segmentation of the undressed model image in the format of a semantic label map; or
(b) wherein the subsequent image training inputs, which provide additional information regarding the skin features and locations, are used to improve the prediction quality of the machine learning system.

### Claim 21

(canceled)

### Claim 22 (depends on 1)

The method of claim 1, (a) wherein for neural network training of the machine learning system, a perception loss based on a weighted-average of convolutional layer responses of a network which classifies images into object categories is adopted; or
(b) wherein the machine learning system includes a machine learning module; or
(c) wherein the machine learning system includes a cascade refinement network (CRN).

### Claim at position 21

23-24. (canceled)

### Claim 25 (depends on 1)

The method of claim 1, wherein the machine learning system uses a cascade refinement network (CRN) containing a cascade of N convolutional layers blocks working at low spatial resolutions to high spatial resolutions, in which lowest resolution version of input image L0 is fed into first block (block 1); feature output Fi of each block i (i=1, 2 . . . N−1) of resolution r are upsampled and then combined with inputs Li+1 scaled at working resolution 2r of block i+1 to be passed as input of next block i+1 in the cascade, and repeated up to last block (block N) of the network, which generates the digital file including photorealistic changes.

### Claim 26 (depends on 25)

The method of claim 25, wherein a channel-stacked image provides unified inputs Li (i=1, 2, . . . N) into the CRN, wherein the channel-stacked image is a consolidated image that combines the garment layer index mask input with the undressed body image and a body mask.

### Claim 27 (depends on 26)

The method of claim 26, (I) wherein two channel-stacked images are used alongside each other:
(a) K+3 channel-stacked image, which includes 3-channel of RGB no-shadow images, K-channel of one-hot encoded garment layer index mask images, wherein all source images are normalized to same size, and
(b) K+8 channel-stacked image, which include 3-channel of the RGB no-shadow images, K-channel of the one-hot encoded garment layer index mask images, 3-channel of the undressed model photos, and a 1-channel binary body mask; or
(II) wherein two channel-stacked images are used alongside each other:
(a) K+3 channel-stacked image, which include 3-channel of RGB no-shadow images, K-channel of one-hot encoded garment layer index mask images, wherein all source images are normalized to same size, and
(b) K+8 channel-stacked image, which include 3-channels of the RGB no-shadow images, K-channels of the one-hot encoded garment layer index mask images, 3-channels of the undressed model photos, and a 1-channel binary body mask generated from an undressed model photo using an automatic image segmentation algorithm; or
(III) wherein for CRN training, the channel-stacked images are scaled to different resolution to be fed as input Li to each block i (i=1, 2, . . . N).

### Claim at position 25

28-29. (canceled)

### Claim 30 (depends on 1)

The method of claim 1, wherein the method uses automated scripts to parse stored data profiles of photo-editing tools, to automatically extract required layers and intermediate image layers from those parsed files, and transform file data to obtain required input-output pairs needed for model training.

### Claim 31 (depends on 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11)

The method of claims 1 to 11 claim 1, wherein the digital images are 3D images.

### Claim 32 (depends on 1)

The method of claim 1, (a) in which the received one or more digital cutout garment textures are obtained from a database; or
(b) in which the received digital image of the dressable model is one in which the dressable model has been separated from a background; or
(c) in which the received one or more digital cutout garment textures require no manipulation.

### Claim at position 29

33-34. (canceled)

### Claim 35

A computer program product executable on a processor to provide a digital file including photorealistic changes for a digital image, the computer program product executable to:
(i) receive a digital image of a dressable model;
(ii) receive one or more digital cutout garment textures, indexed according to an outfitting layering order, aligned with the body shape and pose of the dressable model;
(iii) receive binary silhouettes of the one or more digital cutout garment textures;
(iv) generate a garment layer index mask by compositing the binary silhouettes of the one or more digital cutout garment textures, indexed according to the outfitting layering order;
(v) receive a composite image obtained by overlaying the one or more digital cutout garment textures, according to the indexed outfitting layering order, on the digital image of the dressable model;
(vi) input the composite image and the garment layer index mask into a machine learning system for providing photorealistic changes; and
(vii) receive from the machine learning system a digital file including photorealistic changes, suitable for application to the composite image;
the computer program product further executable to store the digital file including the photorealistic changes.

### Claim at position 31

36-37. (canceled)

### Claim 38

A system including a processor and a machine learning system, the processor configured to provide a digital file including photorealistic changes for a digital image, the processor configured to:
(i) receive a digital image of a dressable model;
(ii) receive one or more digital cutout garment textures, indexed according to an outfitting layering order, aligned with the body shape and pose of the dressable model;
(iii) receive binary silhouettes of the one or more digital cutout garment textures;
(iv) generate a garment layer index mask by compositing the binary silhouettes of the one or more digital cutout garment textures, indexed according to the outfitting layering order;
(v) receive a composite image obtained by overlaying the one or more digital cutout garment textures, according to the indexed outfitting layering order, on the digital image of the dressable model;
(vi) input the composite image and the garment layer index mask into the machine learning system for providing photorealistic changes; and
(vii) receive from the machine learning system a digital file including photorealistic changes, suitable for application to the composite image;
the system further configured to store the digital file including photorealistic changes.

### Claim at position 33

39-55. (canceled)