# US 2022/0019840 A1 · Method, device and apparatus for recognizing, categorizing and searching for garment, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0019840 A1
- Document ID: US20220019840A1
- Status: Published application
- Kind: A1
- Application number: 17/295,337
- Filed: 2019-12-23
- Published: 2022-01-20
- Earliest claimed: 2019-02-18
- Assignees: Beijing Jingdong Shangke Information Technology Co., Ltd.; Beijing Jingdong Century Trading Co., Ltd.
- Applicants: Beijing Jingdong Shangke Information Technology Co., Ltd.; Beijing Jingdong Century Trading Co., Ltd.
- Inventors: Hongbin XIE
- Web page: https://patentgrep.com/patent/US20220019840A1

## CPC

- G06F 18/22: Matching criteria, e.g. proximity measures
- G06Q 30/0282: Rating or review of business operators or products
- G06F 18/211: Selection of the most significant subset of features
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06Q 30/0601: Electronic shopping [e-shopping]
- G06V 10/40: Extraction of image or video features
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/778: Active pattern-learning, e.g. online learning of image or video features
- G06V 10/82: using neural networks
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- Y02P 90/30: Computing systems specially adapted for manufacturing
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- Y02P: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS

## Abstract

A method, device and apparatus for recognizing, categorizing and searching for a garment, and a storage medium. The method for recognizing a garment comprises: acquiring a target image containing a garment to be recognized, and determining, on the basis of the target image, a set of heat maps corresponding to key feature points contained in the target image, the set of heat maps comprising position probability heat maps corresponding to the respective key feature points contained in the target image (101); and processing the set of heat maps on the basis of a shape constraint corresponding to the target image, and determining position probability information of the key feature points contained in the target image (102).

## Claims

### Claim 1

A method for garment recognition, comprising:
acquiring a target image containing a garment to be recognized, and determining, based on the target image, a set of heat maps corresponding to key feature points contained in the target image, the set of heat maps comprising position probability heat maps respectively corresponding to the key feature points contained in the target image; and
processing the set of heat maps based on a shape constraint corresponding to the target image, to determine position probability information of the key feature points contained in the target image.

### Claim 2 (depends on 1)

The method for garment recognition of claim 1, wherein the determining, based on the target image, the set of heat maps corresponding to the key feature points contained in the target image comprises:
processing the target image through a trained first neural network, to determine the set of heat maps corresponding to the key feature points contained in the target image,
and wherein the processing the set of heat maps based on the shape constraint corresponding to the target image, to determine the position probability information of the key feature points contained in the target image comprises:
processing the set of heat maps through a trained second neural network based on the shape constraint corresponding to the target image, to determine the position probability information of the key feature points contained in the target image.

### Claim 3 (depends on 2)

The method for garment recognition of claim 2, wherein the method further comprises: before acquiring the target image containing the garment to be recognized,
acquiring an image training set containing training images of a plurality of garments, the training image comprising an original image carrying marking information of the key feature points; and
training an initial convolutional neural network iteratively based on the image training set, until a loss function meets a convergence condition, to obtain the trained first neural network.

### Claim 4 (depends on 3)

The method for garment recognition of claim 3, wherein the method further comprises: before training the initial convolutional neural network iteratively based on the image training set, performing image augmentation on the original image to obtain a corresponding augmented image, the image training set further comprising the augmented image.

### Claim 5 (depends on 4)

The method for garment recognition of claim 4, wherein the performing image augmentation on the original image to obtain the corresponding augmented image comprises:
performing at least one of image horizontal translation, image vertical translation, color perturbation or image rotation on the original image, to obtain the corresponding augmented image.

### Claim 6 (depends on 3)

The method for garment recognition of claim 3, wherein the method comprises: after obtaining the trained first neural network,
acquiring a set of training samples comprising the set of heat maps corresponding to the key feature points contained in the training images output by the first neural network; and
training an initial bi-directional cyclic convolutional neural network iteratively based on the set of training samples and a set of shape constraints, until a loss function meets a convergence condition to obtain the trained second neural network.

### Claim 7 (depends on 6)

The method for garment recognition of claim 6, wherein the set of shape constraints comprises shape constraints formed by characterizing constraint relationships among a plurality of key feature points by using at least one of triangular substructure or quadrilateral substructure.

### Claim 8 (depends on 6)

The method for garment recognition of claim 6, wherein the method comprises: before processing the set of heat maps through the trained second neural network based on the shape constraint corresponding to the target image,
determining the shape constraint corresponding to the target image according to the set of heat maps corresponding to the key feature points contained in the target image, the target image comprising at least one shape constraint.

### Claim 9 (depends on 6)

The method for garment recognition of claim 6, wherein the training the initial bi-directional cyclic convolutional neural network iteratively based on the set of training samples and the set of shape constraints comprises:
inputting the set of training samples and the set of shape constraints into the initial bi-directional cyclic convolutional neural network; and
training the bi-directional cyclic convolutional neural network iteratively by inputting each shape constraint in the set of shape constraints into a forward layer in a set order and inputting each shape constraint in the set of shape constraints into a backward layer in a set reverse order.

### Claim 10 (depends on 9)

The method for garment recognition of claim 9, wherein training the bi-directional cyclic convolutional neural network iteratively by inputting each shape constraint in the set of shape constraints into the forward layer in the set order and inputting each shape constraint in the set of shape constraints into the backward layer in the set reverse order comprises:
in an iteration, for the bi-directional cyclic convolutional neural network, setting connecting order of the key feature points according to constraint relationships among the key feature points contained in a corresponding shape constraint; and
taking a key feature point contained in the set of heat maps corresponding to the training image as a starting point, inputting position probability heat maps respectively corresponding to the key feature points into the forward layer and the backward layer according to the connecting order and a reverse order of the connecting order at the same time to perform training iteratively

### Claim 11 (depends on 2)

The method for garment recognition of claim 2, wherein the processing the set of heat maps through the trained second neural network based on the shape constraint corresponding to the target image, to determine the position probability information of the key feature points contained in the target image comprises:
adjusting, by the second neural network, a display parameter of the set of heat maps, according to the shape constraint corresponding to the target image; and
adjusting the display parameter of the set of heat maps according to a condition to be met by the display parameter, to obtain a corresponding target parameter, and obtaining the position probability information of the key feature points contained in the target image according to the target parameter.

### Claim 12 (depends on 1)

The method for garment recognition of claim 1, further comprising:
determining a category attribute of the garment based on the position probability information of the key feature points contained in the target image; wherein the category attribute comprises one of a shape, a layout or a style; and
determining a corresponding category of the garment based on the category attribute.

### Claim 13 (depends on 1)

The method for garment recognition of claim 1, further comprising:
determining a category attribute of the garment based on the position probability information of the key feature points contained in the target image; wherein the category attribute comprises one of a shape, a layout or a style;
determining a corresponding search element based on the category attribute; wherein the search element comprises at least one of a search keyword or an image; and
searching a set of garment images corresponding to the target image based on the search element.

### Claim at position 14

-17. (canceled)

### Claim 18

A non-transitory computer storage medium having stored thereon a computer program that when executed by a processor, implements the method for garment recognition, the method comprising:
acquiring a target image containing a garment to be recognized, and determining, based on the target image, a set of heat maps corresponding to key feature points contained in the target image, the set of heat maps comprising position probability heat maps respectively corresponding to the key feature points contained in the target image; and
processing the set of heat maps based on a shape constraint corresponding to the target image, to determine position probability information of the key feature points contained in the target image.

### Claim 19

A device for garment recognition, comprising:
a processor and a memory configured to store a computer program executable on the processor;
wherein the processor is configured to execute the computer program to implement:
acquiring a target image containing a garment to be recognized, and determining, based on the target image, a set of heat maps corresponding to key feature points contained in the target image, the set of heat maps comprising position probability heat maps respectively corresponding to the key feature points contained in the target image; and
processing the set of heat maps based on a shape constraint corresponding to the target image, to determine position probability information of the key feature points contained in the target image.

### Claim 20 (depends on 19)

The device for garment recognition of claim 19, wherein the determining, based on the target image, the set of heat maps corresponding to the key feature points contained in the target image comprises:
processing the target image through a trained first neural network, to determine the set of heat maps corresponding to the key feature points contained in the target image,
and wherein the processing the set of heat maps based on the shape constraint corresponding to the target image, to determine the position probability information of the key feature points contained in the target image comprises:
processing the set of heat maps through a trained second neural network based on the shape constraint corresponding to the target image, to determine the position probability information of the key feature points contained in the target image.

### Claim 21 (depends on 20)

The device for garment recognition of claim 20, wherein the processor is configured to execute the computer program to further implement: before acquiring the target image containing the garment to be recognized, acquiring an image training set containing training images of a plurality of garments, the training image comprising an original image carrying marking information of the key feature points; and
training an initial convolutional neural network iteratively based on the image training set, until a loss function meets a convergence condition, to obtain the trained first neural network.

### Claim 22 (depends on 21)

The device for garment recognition of claim 21, wherein the processor is configured to execute the computer program to further implement: before training the initial convolutional neural network iteratively based on the image training set, performing image augmentation on the original image to obtain a corresponding augmented image, the image training set further comprising the augmented image.

### Claim 23 (depends on 22)

The device for garment recognition of claim 22, wherein the performing image augmentation on the original image to obtain the corresponding augmented image comprises:
performing at least one of image horizontal translation, image vertical translation, color perturbation or image rotation on the original image, to obtain the corresponding augmented image.

### Claim 24 (depends on 21)

The device for garment recognition of claim 21, wherein the processor is configured to execute the computer program to further implement: after obtaining the trained first neural network, acquiring a set of training samples comprising the set of heat maps corresponding to the key feature points contained in the training images output by the first neural network; and
training an initial bi-directional cyclic convolutional neural network iteratively based on the set of training samples and a set of shape constraints, until a loss function meets a convergence condition to obtain the trained second neural network.
