# US 2020/0320769 A1 · Method and system for predicting garment attributes using deep learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0320769 A1
- Document ID: US20200320769A1
- Status: Published application
- Kind: A1
- Application number: 16/304,380
- Filed: 2017-05-25
- Published: 2020-10-08
- Earliest claimed: 2016-05-25
- Assignees: Metail Limited
- Applicants: Metail Limited
- Inventors: Yu CHEN; Sukrit SHANKAR; Jim DOWNING; Joe TOWNSEND; Duncan ROBERTSON; Tom ADEYOOLA
- Web page: https://patentgrep.com/patent/US20200320769A1

## CPC

- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06F 16/535: Filtering based on additional data, e.g. user or group profiles
- G06F 16/538: Presentation of query results
- G06F 16/5838: {using colour}
- G06F 16/5862: {using texture}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/24: Classification techniques
- G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30)
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06N 3/096: Transfer learning
- G06T 15/005: {General purpose rendering architectures}
- G06T 15/04: Texture mapping
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/64: Three-dimensional [3D] objects
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V 2201/12: Acquisition of 3D measurements of objects
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

There is provided a computer implemented method for predicting garment or accessory attributes using deep learning techniques, comprising the steps of: (i) receiving and storing one or more digital image datasets including images of garments or accessories; (ii) training a deep model for garment or accessory attribute identification, using the stored one or more digital image datasets, by configuring a deep neural network model to predict (a) multiple-class discrete attributes; (b) binary discrete attributes, and (c) continuous attributes, (iii) receiving one or more digital images of a garment or an accessory, and (iv) extracting attributes of the garment or the accessory from the one or more received digital images using the trained deep model for garment or accessory attribute identification. A related system is also provided.

## Claims

### Claim 1

A computer-implemented method for predicting garment or accessory attributes using deep learning techniques, comprising the steps of:
(i) receiving and storing one or more digital image datasets including images of garments or accessories;
(ii) training a deep model for garment or accessory attribute identification, using the stored one or more digital image datasets, by configuring a deep neural network model to predict
(a) multiple-class discrete attributes;
(b) binary discrete attributes, and
(c) continuous attributes,
(iii) receiving one or more digital images of a garment or an accessory, and
(iv) extracting attributes of the garment or the accessory from the one or more received digital images using the trained deep model for garment or accessory attribute identification.

### Claim 2

(canceled)

### Claim 3 (depends on 1)

The method of claim 1, wherein the extracted attributes include one or more of: style, shape, texture, colour, fabric properties.

### Claim 4 (depends on 1)

The method of claim 1, wherein the one or more digital image datasets include a digital image dataset based on garment images.

### Claim 5 (depends on 1)

The method of claim 1, the method including the step of: generating annotations for the digital image dataset based on garment images using natural language processing, and storing the generated annotations in the digital image dataset based on garment images.

### Claim 6 (depends on 4)

The method of claim 4, wherein the digital image dataset based on garment images is a digital image dataset based on internet garment images.

### Claim 7 (depends on 4)

The method of claim 4, wherein the one or more digital image datasets includes a digital image dataset based on sets of garment mannequin photos which includes metadata and multiple semantic labels associated with sets of garment mannequin photos.

### Claim 8 (depends on 7)

The method of claim 7, wherein the digital image dataset based on sets of garment mannequin photos includes digital images of garments taken on the mannequin in a controlled lighting environment, in a standard camera pose.

### Claim 9 (depends on 7)

The method of claim 7, wherein the digital image dataset based on sets of garment mannequin photos includes high-resolution unsegmented original photos of the garment samples and segmented garment texture sprites, both in 8 distinct camera views.

### Claim 10 (depends on 7)

The method of claim 7, wherein the metadata and multiple semantic labels associated with sets of garment mannequin photos include one or more of: Garment name and description; Garment category and subcategory; Colour; Pattern and texture; Fit styles; Vertical drops; Fabric and material composition; Washing method; Price or price range.

### Claim 11 (depends on 7)

The method of claim 7, wherein regarding the digital image dataset based on sets of garment mannequin photos, keyword extraction or natural language processing (NLP) is used to extract style-related attributes and semantic labels from the garment name and garment description text.

### Claim 12 (depends on 7)

The method of claim 7, wherein regarding the digital image dataset based on sets of garment mannequin photos, metadata and/or semantic labels are structured 1) by associating groups of different keywords of similar meanings, and/or 2) assigning label weights with values in a range.

### Claim 13 (depends on 1)

The method of claim 1, wherein the digital image datasets include one or more of: unsegmented mannequin photos of a garment, either in a single frontal view, or in multiple distinct camera views; segmented garment texture sprites from mannequin photos; sample photos of a garment on a retailer's website; and synthetic garment images obtained by rendering a simulated garment model using computer graphic techniques.

### Claim at position 14

14-20. (canceled)

### Claim 21 (depends on 1)

The method of claim 1, wherein the step of training the deep model for garment or accessory attribute identification is such that a combination of multiple discrete and continuous attributes are modelled simultaneously.

### Claim at position 16

22-33. (canceled)

### Claim 34 (depends on 1)

The method of claim 1, the method including a computer-implemented method of digitising a garment, and estimating the physics parameters of the garment fabric material, using a garment digitization apparatus, the apparatus including a mannequin, a mannequin rotation system, a computer system and a camera system, the method including the steps of:
(i) imaging a mannequin wearing a garment using the camera system;
(ii) rotating the mannequin wearing the garment through at least 360° using the mannequin rotation system;
(iii) capturing at least three images of the garment using the camera system during the mannequin rotation,
(iv) generating fast and jerky left-right-left rotations at a series of configured rotational accelerations and velocities to disturb the garment on the mannequin with patterned motion, and
(v) capturing the garment appearance under motion and estimating the physics parameters of the garment fabric material.

### Claim 35 (depends on 34)

The method of claim 34, wherein multiple images of the target garment are photographed at scheduled times during the course of the vibration sequence to capture the appearance of the garment under different stages of the motion, wherein the images include (a) at least one image capturing the static status of the target garment, and (b) one or more images capturing the target garment under motion.

### Claim 36 (depends on 34)

The method of claim 34, including analyzing captured garment images in different phases of garment motion and predicting the garment fabric properties and/or model parameters for garment physics simulation.

### Claim 37 (depends on 36)

The method of claim 36, including the step of storing the predicted physics parameters into a garment database.

### Claim 38 (depends on 34)

The method of claim 34, including the step of using a mannequin with a pressure sensor array embedded on or under the surface of the mannequin, which captures the stress/strain of the garment when the garment is dressed on the mannequin.

### Claim 39 (depends on 34)

The method of claim 34, including the steps of capturing the garment appearance under motion, measuring the strain and stretch of the garment when dressed on the mannequin, and estimating the physical parameters of the garment fabric material, and using the estimated physical parameters of the garment fabric material for photo-realistic and dynamic garment simulation and rendering in the application of virtual fitting.

### Claim 40 (depends on 1)

The method of claim 1, the method further including a computer-implemented method to improve the photo-realism of a rendered body image or a virtual avatar image.

### Claim at position 24

41-42. (canceled)

### Claim 43 (depends on 1)

The method of claim 1, the method further including a computer-implemented method to evaluate the level of photo-realism of synthetic renders of body images against real photos.

### Claim 44 (depends on 40)

The method of claim 40, which includes the steps of:
i) collecting one or more real photos and one or more synthetic rendered images as positive and negative samples,
ii) training a machine learning model to generate a difference image,
iii) using the machine learning model to generate a difference image,
iv) superposing the difference images onto the input synthetic rendered image to generate a more photo-realistic synthetic image.

### Claim 44 (depends on 44)

The method of claim 44. wherein training and using the machine learning model includes the step of using two adversarial submodules: the first submodule distinguishes the synthetic virtual avatar renders from real photos of models wearing garments, and the second submodule makes modifications to the initial render output and aims to improve the photo-realism of synthetic renders of body image.

### Claim 45 (depends on 44)

The method of claim 44, wherein the machine learning model is a deep neural network.

### Claim 47 (depends on 1)

The method of claim 1, in which the one or more digital photos of a garment or an accessory are received in a query, with the goal of finding similar items to the queried item.

### Claim 48 (depends on 1)

The method of claim 1, in which the one or more digital photos of a garment or an accessory are received in a query, with the goal of identifying the item provided in the query.

### Claim 49 (depends on 47)

The method of claim 47, wherein the one or more digital photos are of items which are one or more of: currently dressed on user's virtual avatar; recently browsed by the user; in an arbitrary photo on the internet.

### Claim 50 (depends on 1)

The method of claim 1, in which an attribute-based search is provided, in which an input is a set of keywords describing the query item.

### Claim 51 (depends on 1)

The method of claim 1, in which an approach for image-based search and image retrieval is: (a) obtaining the extracted attributes of the garment or accessory, (b) computing the feature distances between a query image and each image in the digital image datasets using a distance metric based on the extracted attributes of the garment or accessory; (c) presenting the search or retrieval results by ranking, using the computed distance metrics.

### Claim at position 34

52-54. (canceled)

### Claim 55 (depends on 1)

The method of claim 1, in which at the input side of the image-based garment search and retrieval system an ROI Detection Module is included, which detects the region-of-interest (ROI) of the garment in the form of bounding boxes on both the input query image and all the gallery images as a pre-processsing step.

### Claim 56 (depends on 55)

The method of claim 55, in which multiple bounding boxes each surrounding an individual garment or accessory item are provided as a pre-processsing step.

### Claim at position 37

57-61. (canceled)

### Claim 62 (depends on 1)

The method of claim 1, including, in a size advice and fit analysis, receiving user information, including one or more of, user's body shape parameters, user's location, age, and ethnicity;
receiving garment sizing and measurement information, including one or more of: garment sizes, size-charts of the garment, garment measurements on the QC sheets;
receiving garment images and fit-style labels, including one or more of: the circumferential fits over different body parts and vertical drops, and
including a step of re-architecting and fine-tuning the pre-trained deep model.

### Claim 63 (depends on 62)

The method of claim 62, in which the re-architected model maintains all the convolutional-layers of the pre-trained model but completely rebuilds the original fully-connected (FC) layers.

### Claim 64 (depends on 63)

The method of claim 63, in which the input to the new model includes both a garment mannequin photo and the user features, and the output of the model are 3D size-chart feature vectors.

### Claim 65 (depends on 62)

The method of claim 62, wherein in the fine-tuning process, different layers of the model are re-trained with different learning rates; the weights of the new FC layers are trained at a learning rate 10-times higher than those applied to the weights of the existing convolutional layers, in which the fine-tuning scheme adapts the pre-trained features to the new training data for the size recommendation problem.

### Claim 66 (depends on 62)

The method of claim 62, including a preprocessing step to map all the size labels in the training data into the size-chart feature vectors.

### Claim 67 (depends on 62)

The method of claim 62, wherein multi-image input is provided, in which multiple photos in distinct camera views are available for the target garment.

### Claim 68 (depends on 62)

The method of claim 62, including a size regression algorithm step which is to look up the size feature on the target size-chart and recommend the most similar size.

### Claim 69 (depends on 62)

The method of claim 62, wherein the output of the deep model is simply the size label, which is a multi-class discrete label instead of a continuous label, and a “softmax” activation is applied after the final FC layer to convert the network output into a sum-to-one probability vector.

### Claim 70 (depends on 1)

The method of claim 1, further comprising a method of garment size and fit recommendation, which includes the steps of
i) predicting multiple fit-style labels and associated probabilities of a garment from one or more input photos, including one or more of circumferential fits over different body parts and vertical drops;
ii) selecting a subset of most relevant fit points by thresholding the associated probabilities obtained in i);
iii) predicting the optimal garment size and performing a fit analysis by analysing user measurements and garment measurements over the selected fit points obtained in ii);
iv) providing a fit recommendation.

### Claim 71

System A system for predicting garment or accessory attributes using deep learning techniques, the system including a processor configured to:
(i) receive and store one or more digital image datasets including images of garments or accessories;
(ii) train a deep model for garment or accessory attribute identification, using the stored one or more digital image datasets, by configuring a deep neural network model to predict
(a) multiple-class discrete attributes;
(b) binary discrete attributes, and
(c) continuous attributes,
(iii) receive one or more digital images of a garment or an accessory, and
(iv) extract attributes of the garment or the accessory from the one or more received digital images using the trained deep model for garment or accessory attribute identification.

### Claim at position 48

72-73. (canceled)

### Claim 74

A computer-implemented method of garment size and fit recommendation, which includes the steps of:
i) predicting multiple fit-style labels and associated probabilities of a garment from one or more input photos, including one or more of circumferential fits over different body parts and vertical drops;
ii) selecting a subset of most relevant fit points by thresholding the associated probabilities obtained in i);
iii) predicting the optimal garment size and performing a fit analysis by analysing user measurements and garment measurements over the selected fit points obtained in ii);
iv) providing a fit recommendation.

### Claim 75

(canceled)

### Claim 76

A computer-implemented method of digitising a garment, and estimating the physics parameters of the garment fabric material, the method using a garment digitization apparatus, the apparatus including a mannequin, a mannequin rotation system, a computer system and a camera system, the method including the steps of:
(i) imaging a mannequin wearing a garment using the camera system;
(ii) rotating the mannequin wearing the garment through at least 360° using the mannequin rotation system;
(iii) capturing at least three images of the garment using the camera system during the mannequin rotation,
(iv) generating fast and jerky left-right-left rotations at a series of configured rotational accelerations and velocities to disturb the garment on the mannequin with patterned motion, and
(v) capturing the garment appearance under motion and estimating the physics parameters of the garment fabric material.

### Claim 77 (depends on 76)

The method of claim 76, wherein multiple images of the target garment are photographed at scheduled times during the course of the vibration sequence to capture the appearance of the garment under different stages of the motion, wherein the images include (a) at least one image capturing the static status of the target garment, and (b) one or more images capturing the target garment under motion.

### Claim 78 (depends on 76)

The method of claim 76, including analyzing captured garment images in different phases of garment motion and predicting the garment fabric properties and/or model parameters for garment physics simulation.

### Claim 79 (depends on 78)

The method of claim 78, including the step of storing the predicted physics parameters into a garment database.

### Claim 80

(canceled)

### Claim 81 (depends on 76)

The method of claim 76, including the steps of capturing the garment appearance under motion, measuring the strain and stretch of the garment when dressed on the mannequin, and estimating the physical parameters of the garment fabric material, and using the estimated physical parameters of the garment fabric material for photo-realistic and dynamic garment simulation and rendering in the application of virtual fitting.

### Claim 82

A system for digitising a garment, and estimating the physics parameters of the garment fabric material, the system including a garment digitization apparatus, the apparatus including a mannequin, a mannequin rotation system, a computer system and a camera system, the system arranged to:
(i) image a mannequin wearing a garment using the camera system;
(ii) rotate the mannequin wearing the garment through at least 360° using the mannequin rotation system;
(iii) capture at least three images of the garment using the camera system during the mannequin rotation,
(iv) generate fast and jerky left-right-left rotations at a series of configured rotational accelerations and velocities to disturb the garment on the mannequin with patterned motion, and
(v) capture the garment appearance under motion and estimate the physics parameters of the garment fabric material.

### Claim 83

(canceled)