# US 11,270,476 B2 · Method and system for providing photorealistic changes for digital image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,270,476 B2
- Document ID: US11270476B2
- Status: Granted
- Kind: B2
- Application number: 17/050,587
- Filed: 2019-04-23
- Published: 2021-08-05
- Granted: 2022-03-08
- Earliest claimed: 2018-04-24
- Assignees: Metail Limited
- Applicants: Metail Limited
- Inventors: Yu Chen; Jim Downing; Tom Adeyoola; Sukrit Shankar
- Web page: https://patentgrep.com/patent/US11270476B2

## CPC

- G06T 11/00: Two-dimensional [2D] image generation
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 20/00: Machine learning
- G06Q 30/0643: {graphically representing goods, e.g. 3D product representation}
- G06T 3/18: Image warping, e.g. rearranging pixels individually
- G06T 7/11: Region-based segmentation
- G06T 7/194: involving foreground-background segmentation
- G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77)
- G06T 2207/20081: Training; Learning
- G06T 2210/16: Cloth
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Disclosed is a computer-implemented method for providing photorealistic changes for a digital image. The method includes receiving a digital image of dressable model, receiving digital cutout garment textures that are indexed according to an outfitting layering order and aligned with body shape and pose of the dressable model, receiving binary silhouettes of the digital cutout garment textures, generating a garment layer index mask by compositing the binary silhouettes of the digital cutout garment textures indexed according to the outfitting layering order, receiving a composite image obtained by overlaying the digital cutout garment textures according to the indexed outfitting layering order on the digital image of the dressable model, inputting the composite image and the garment layer index mask into a machine learning system for providing photorealistic changes, and receiving from the machine learning system a digital file including photorealistic changes for application to the composite image.

## Claims

### Claim 1

A computer-implemented method for providing photorealistic changes for a digital image, the method including steps of:
(i) receiving a digital image of a dressable model;
(ii) receiving one or more digital cutout garment textures, indexed according to an outfitting layering order, aligned with a body shape and a pose of the dressable model;
(iii) receiving binary silhouettes of the one or more digital cutout garment textures;
(iv) generating a garment layer index mask by compositing the binary silhouettes of the one or more digital cutout garment textures, indexed according to the outfitting layering order;
(v) receiving a composite image obtained by overlaying the one or more digital cutout garment textures, according to the indexed outfitting layering order, on the digital image of the dressable model;
(vi) inputting the composite image and the garment layer index mask into a machine learning system for providing photorealistic changes; and
(vii) receiving from the machine learning system a digital file including photorealistic changes, suitable for application to the composite image;
the method including the step of storing the digital file including the photorealistic changes.

### Claim 2 (depends on 1)

The method of claim 1, wherein the step (i) includes generating or receiving a binary mask of the digital image of the dressable model, and wherein step (vi) includes inputting the binary mask of the digital image of the dressable model into the machine learning system for providing photorealistic changes.

### Claim 3 (depends on 1)

The method of claim 1, including a step in which the received composite image is generated by overlaying the one or more digital cutout garment textures, according to the indexed outfitting layering order, on the digital image of the dressable model.

### Claim 4 (depends on 1)

The method of claim 1, including steps in which
(a) each of the one or more digital cutout garment textures is generated by separating or cutting out a garment in a foreground of the image from a garment image background, and then
(b) is transformed or warped to match or align with the body shape and pose of the model.

### Claim 5 (depends on 1)

The method of claim 1, including a step of generating a quality-enhanced composed image by applying the digital file including photorealistic changes, to the composite image.

### Claim 6 (depends on 1)

The method of claim 1, in which the photorealistic changes are one or more of: a shadow or shadows, a crease or creases, a scene background or scene backgrounds, a scene setting or scene settings, an animation or animations, or lighting adaption.

### Claim 7 (depends on 1)

The method of claim 1, including a step in which the machine learning system is trained.

### Claim 8 (depends on 7)

The method of claim 7, (a) including a step in which to train the machine learning system at least 1,000 images are used; or
(b) in which the step in which the machine learning system is trained includes a multi-stage iterative training process wherein the machine learning system is initially trained using a model for low-resolution-images, and then is gradually re-trained or fine tuned for greater image resolutions through weight-sharing to produce a training model.

### Claim 9 (depends on 1)

The method of claim 1, wherein the digital images are 2D images.

### Claim 10 (depends on 1)

The method of claim 1, (a) wherein the method generates ground truth diff shadow image by subtracting a final composed image from an intermediate composed image obtained after overlaying one or more digital cutout garment textures, according to indexed outfitting layering order, on digital image of dressable model, for training the machine learning system; or
(b) wherein the digital image of a dressable model is selected based on a predetermined selection criterion or criteria; or
(c) wherein each binary silhouette is generated by extracting and thresholding alpha channel of a respective digital cutout garment texture; or
(d) in which background regions and regions containing the undressed model or any garments or accessories within the undressed model image are all indexed with the lowest layering order index value.

### Claim 11 (depends on 1)

The method of claim 1, wherein the garment layer index mask conveys important spatial information that guides the machine learning system on how and where to precisely render photo realistic features.

### Claim 12 (depends on 11)

The method of claim 11, wherein the garment layer index mask conveys important spatial information that guides the machine learning system on, in which direction such photo realistic features should be added.

### Claim 13 (depends on 1)

The method of claim 1, wherein subsequent image training inputs are used to improve prediction quality of the machine learning system.

### Claim 14 (depends on 13)

The method of claim 13, (a) wherein the subsequent image training inputs include one or more of: an undressed model image; an undressed model image in format of a three channel RGB image; a body mask including a binary foreground and background segmentation of the undressed model image; a body mask including a binary foreground and background segmentation of the undressed model image in the format of a semantic label map; or
(b) wherein the subsequent image training inputs, which provide additional information regarding the skin features and locations, are used to improve the prediction quality of the machine learning system.

### Claim 15 (depends on 1)

The method of claim 1, (a) wherein for neural network training of the machine learning system, a perception loss based on a weighted-average of convolutional layer responses of a network which classifies images into object categories is adopted; or
(b) wherein the machine learning system includes a machine learning module; or
(c) wherein the machine learning system includes a cascade refinement network (CRN).

### Claim 16 (depends on 1)

The method of claim 1, wherein the machine learning system uses a cascade refinement network (CRN) containing a cascade of N convolutional layers blocks working at low spatial resolutions to high spatial resolutions, in which lowest resolution version of input image L0 is fed into first block (block 1); feature output Fi of each block i (i=1, 2 . . . N−1) of resolution r are upsampled and then combined with inputs Li+1 scaled at working resolution 2r of block i+1 to be passed as input of next block i+1 in the cascade, and repeated up to last block (block N) of the network, which generates the digital file including photorealistic changes.

### Claim 17 (depends on 16)

The method of claim 16, wherein a channel-stacked image provides unified inputs Li (i=1, 2, . . . N) into the CRN, wherein the channel-stacked image is a consolidated image that combines the garment layer index mask input with the undressed body image and a body mask.

### Claim 18 (depends on 17)

The method of claim 17, (I) wherein two channel-stacked images are used alongside each other:
(a) K+3 channel-stacked image, which includes 3-channel of RGB no-shadow images, K-channel of one-hot encoded garment layer index mask images, wherein all source images are normalized to same size, and
(b) K+8 channel-stacked image, which include 3-channel of the RGB no-shadow images, K-channel of the one-hot encoded garment layer index mask images, 3-channel of the undressed model photos, and a 1-channel binary body mask; or
(II) wherein two channel-stacked images are used alongside each other:
(a) K+3 channel-stacked image, which include 3-channel of RGB no-shadow images, K-channel of one-hot encoded garment layer index mask images, wherein all source images are normalized to same size, and
(b) K+8 channel-stacked image, which include 3-channels of the RGB no-shadow images, K-channels of the one-hot encoded garment layer index mask images, 3-channels of the undressed model photos, and a 1-channel binary body mask generated from an undressed model photo using an automatic image segmentation algorithm; or
(III) wherein for CRN training, the channel-stacked images are scaled to different resolution to be fed as input Li to each block i (i=1, 2, . . . N).

### Claim 19 (depends on 1)

The method of claim 1, wherein the method uses automated scripts to parse stored data profiles of photo-editing tools, to automatically extract required layers and intermediate image layers from those parsed files, and transform file data to obtain required input-output pairs needed for model training.

### Claim 20 (depends on 1)

The method of claim 1, wherein the digital images are 3D images.

### Claim 21 (depends on 1)

The method of claim 1, (a) in which the received one or more digital cutout garment textures are obtained from a database; or
(b) in which the received digital image of the dressable model is one in which the dressable model has been separated from a background; or
(c) in which the received one or more digital cutout garment textures require no manipulation.

### Claim 22

A computer program product comprising a non-transitory computer-readable storage medium, and is executable on a processor to provide a digital file including photorealistic changes for a digital image, the computer program product executable to:
(i) receive a digital image of a dressable model;
(ii) receive one or more digital cutout garment textures, indexed according to an outfitting layering order, aligned with the body shape and pose of the dressable model;
(iii) receive binary silhouettes of the one or more digital cutout garment textures;
(iv) generate a garment layer index mask by compositing the binary silhouettes of the one or more digital cutout garment textures, indexed according to the outfitting layering order;
(v) receive a composite image obtained by overlaying the one or more digital cutout garment textures, according to the indexed outfitting layering order, on the digital image of the dressable model;
(vi) input the composite image and the garment layer index mask into a machine learning system for providing photorealistic changes; and
(vii) receive from the machine learning system a digital file including photorealistic changes, suitable for application to the composite image;
the computer program product further executable to store the digital file including the photorealistic changes.

### Claim 23

A system including a processor and a machine learning system, the processor configured to provide a digital file including photorealistic changes for a digital image, the processor configured to:
(i) receive a digital image of a dressable model;
(ii) receive one or more digital cutout garment textures, indexed according to an outfitting layering order, aligned with the body shape and pose of the dressable model;
(iii) receive binary silhouettes of the one or more digital cutout garment textures;
(iv) generate a garment layer index mask by compositing the binary silhouettes of the one or more digital cutout garment textures, indexed according to the outfitting layering order;
(v) receive a composite image obtained by overlaying the one or more digital cutout garment textures, according to the indexed outfitting layering order, on the digital image of the dressable model;
(vi) input the composite image and the garment layer index mask into the machine learning system for providing photorealistic changes; and
(vii) receive from the machine learning system a digital file including photorealistic changes, suitable for application to the composite image;
the system further configured to store the digital file including photorealistic changes.