# US 11,443,468 B2 · Training and using an ensemble of complimentary convolutional neural networks for cross-domain retrieval of fashion item images > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,468 B2 - Document ID: US11443468B2 - Status: Granted - Kind: B2 - Application number: 16/808,483 - Filed: 2020-03-04 - Published: 2021-09-09 - Granted: 2022-09-13 - Earliest claimed: 2020-03-04 - Assignees: Naver Corporation - Applicants: Naver Corporation - Inventors: Naila Murray; Michal Kucer - Web page: https://patentgrep.com/patent/US11443468B2 ## CPC - G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215) - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 11/60: Creating or editing images; Combining images with text - G06V 10/764: using classification, e.g. of video objects - G06V 10/776: Validation; Performance evaluation - G06V 10/82: using neural networks - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2210/22: Cropping - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method and system generate an ensemble image representation for cross-domain retrieval of a fashion item image from a database by using a three-stream Siamese triplet loss trained convolutional neural network to generate a first retrieval descriptor corresponding to an inputted query image; using an average precision loss trained convolutional neural network to generate a second retrieval descriptor corresponding to the inputted query image; concatenating both the first retrieval descriptor and the second retrieval descriptor; and I2-normalizing the concatenated result to generate the ensemble image representation. During a first stage of the method and system, database items are cropped using a trained fine-grained fashion item detector. ## Claims ### Claim 1 A network ensemble for generating an ensemble image representation for cross-domain retrieval of a fashion item image from a database, the network ensemble accompanied by a processor and a memory, comprising: a three-stream Siamese triplet loss trained convolutional neural network for generating a first retrieval descriptor corresponding to an inputted query image; an average precision loss trained convolutional neural network for generating a second retrieval descriptor corresponding to the inputted query image; and a fusion layer, operatively connected to said three-stream Siamese triplet loss trained convolutional neural network and said average precision loss trained convolutional neural network, for generating an ensemble image representation by concatenating both said first retrieval descriptor and said second retrieval descriptor and I2-normalizing the concatenated result; said three-stream Siamese triplet loss trained convolutional neural network being trained by taking a plurality of triplets of fashion item images, where each triplet contains a fashion item query image, a positive fashion item image corresponding to the same item as in the fashion item query image, and a negative fashion item image corresponding to a different item as in the fashion item query image; computing the triplet loss for each of the triplets; ranking the triplets by the triplet loss computed; selecting a subset of triplets among the plurality of triplets, the subset of triplets having the largest computed loss among the plurality of triplets; and retraining the pre-trained convolutional neural network on each of the triplets of the subset of triplets, using backpropagation, to determine trained values of a plurality of parameters of the convolutional neural network. ### Claim 2 (depends on 1) The network ensemble of claim 1, wherein said average precision loss trained convolutional neural network is trained by computing descriptors of all batch fashion item images and discarding intermediary tensors; computing the score matrix S, the mAPQ loss, and the gradient of the loss with respect to the descriptors; and for a given fashion item image in the batch, computing a descriptor, storing the intermediary tensors, and using the computed gradient for the descriptor to continue the backpropagation through the convolutional neural network. ### Claim 3 (depends on 1) The network ensemble of claim 1, wherein fashion item images in the database are cropped. ### Claim 4 (depends on 1) The network ensemble of claim 1, wherein said three-stream Siamese triplet loss trained convolutional neural network is trained using cropped fashion item images. ### Claim 5 (depends on 2) The network ensemble of claim 2, wherein said average precision loss trained convolutional neural network is trained using cropped fashion item images. ### Claim 6 A method for generating an ensemble image representation for cross-domain retrieval of a fashion item image from a database, comprising: (a) training a three-stream Siamese triplet loss trained convolutional neural network by taking a plurality of triplets of fashion item images, where each triplet contains a fashion item query image, a positive fashion item image corresponding to the same item as in the fashion item query image, and a negative fashion item image corresponding to a different item as in the fashion item query image; computing the triplet loss for each of the triplets; ranking the triplets by the triplet loss computed; selecting a subset of triplets among the plurality of triplets, the subset of triplets having the largest computed loss among the plurality of triplets; and retraining the pre-trained convolutional neural network on each of the triplets of the subset of triplets, using back propagation, to determine trained values of a plurality of parameters of the convolutional neural network; (b) using the three-stream Siamese triplet loss trained convolutional neural network to generate a first retrieval descriptor corresponding to an inputted query image; (c) using an average precision loss trained convolutional neural network to generate a second retrieval descriptor corresponding to the inputted query image; (d) concatenating both the first retrieval descriptor and the second retrieval descriptor; and (e) I2-normalizing the concatenated result to generate the ensemble image representation. ### Claim 7 (depends on 6) The method of claim 6, wherein the average precision loss trained convolutional neural network is trained by computing descriptors of all batch fashion item images and discarding intermediary tensors; computing the score matrix S, the mAPQ loss, and the gradient of the loss with respect to the descriptors; and for a given fashion item image in the batch, computing a descriptor, storing the intermediary tensors, and using the computed gradient for the descriptor to continue the backpropagation through the convolutional neural network. ### Claim 8 (depends on 6) The method of claim 6, wherein fashion item images in the database are cropped. ### Claim 9 (depends on 6) The method of claim 6, wherein the three-stream Siamese triplet loss trained convolutional neural network is trained using cropped fashion item images. ### Claim 10 (depends on 7) The method of claim 7, wherein the average precision loss trained convolutional neural network is trained using cropped fashion item images. ### Claim 11 A system for generating an ensemble image representation for cross-domain retrieval of a fashion item image from a database comprising: one or more processors; and a memory including a network ensemble; said network ensemble including, a three-stream Siamese triplet loss trained convolutional neural network for generating a first retrieval descriptor corresponding to an inputted query image, an average precision loss trained convolutional neural network for generating a second retrieval descriptor corresponding to the inputted query image, and a fusion layer, operatively connected to said three-stream Siamese triplet loss trained convolutional neural network and said average precision loss trained convolutional neural network, for generating an ensemble image representation by concatenating both said first retrieval descriptor and said second retrieval descriptor and I2-normalizing the concatenated result; said average precision loss trained convolutional neural network being trained by computing descriptors of all batch fashion item images and discarding intermediary tensors; computing the score matrix S, the mAPQ loss, and the gradient of the loss with respect to the descriptors; and for a given fashion item image in the batch, computing a descriptor, storing the intermediary tensors, and using the computed gradient for the descriptor to continue the backpropagation through the convolutional neural network. ### Claim 12 (depends on 11) The system of claim 11, wherein said three-stream Siamese triplet loss trained convolutional neural network is trained by taking a plurality of triplets of fashion item images, where each triplet contains a fashion item query image, a positive fashion item image corresponding to the same item as in the fashion item query image, and a negative fashion item image corresponding to a different item as in the fashion item query image; computing the triplet loss for each of the triplets; ranking the triplets by the triplet loss computed; selecting a subset of triplets among the plurality of triplets, the subset of triplets having the largest computed loss among the plurality of triplets; and retraining the pre-trained convolutional neural network on each of the triplets of the subset of triplets, using backpropagation, to determine trained values of a plurality of parameters of the convolutional neural network. ### Claim 13 (depends on 11) The system of claim 11, wherein fashion item images in the database are cropped. ### Claim 14 (depends on 12) The system of claim 12, wherein said three-stream Siamese triplet loss trained convolutional neural network is trained using cropped fashion item images. ### Claim 15 (depends on 11) The system of claim 11, wherein said average precision loss trained convolutional neural network is trained using cropped fashion item images.