# US 2019/0171904 A1 · Method and apparatus for training fine-grained image recognition model, fine-grained image recognition method and apparatus, and storage mediums

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0171904 A1
- Document ID: US20190171904A1
- Status: Published application
- Kind: A1
- Application number: 16/205,005
- Filed: 2018-11-29
- Published: 2019-06-06
- Earliest claimed: 2017-12-01
- Assignees: Baidu Online Network Technology (Beijing) Co., Ltd.
- Applicants: Baidu Online Network Technology (Beijing) Co., Ltd.
- Inventors: Ming Sun; Yuchen Yuan; Feng Zhou
- Web page: https://patentgrep.com/patent/US20190171904A1

## CPC

- G06N 3/08: Learning methods
- G06V 10/82: using neural networks
- G06F 18/211: Selection of the most significant subset of features
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/24143: {Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]}
- G06F 18/2431: Multiple classes
- G06F 18/285: {Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 20/35: {Categorising the entire scene, e.g. birthday party or wedding scene}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The present disclosure provides a method and apparatus for training a fine-grained image recognition model, a device and a storage medium. The method comprises: obtaining images as training samples, and respectively obtaining a tag corresponding to each image, the tag including a class to which the image belongs; training according to the training samples and corresponding tags to obtain a fine-grained image recognition model, and performing constraint at a feature level from two dimensions, namely, the class and object parts, during the training, so that the fine-grained image recognition model learns key object parts in the images; upon performing the fine-grained image recognition, inputting a to-be-recognized image to the fine-grained image recognition model, so that the fine-grained image recognition model positions key object parts in the image, and completes fine-grained image classification according to the key object parts, and outputs a classification result. The solution of the present disclosure can be applied to save manpower costs and improve the model training efficiency.

## Claims

### Claim 1

A method for training a fine-grained image recognition model, wherein the method comprises:
obtaining images as training samples, and respectively obtaining a tag corresponding to each image, the tag including a class to which the image belongs;
training according to the training samples and corresponding tags to obtain a fine-grained image recognition model, and performing constraint at a feature level from two dimensions, namely, the class and object parts, during the training, so that the fine-grained image recognition model learns key object parts in the images, and completes fine-grained image classification according to the key object parts.

### Claim 2 (depends on 1)

The method according to claim 1, wherein
the performing constraint at a feature level from two dimensions, namely, the class and object parts, comprises:
a distance between two features extracted from the same object part in the images in the same class simultaneously satisfies: smaller than a distance between two features extracted from different object parts in the images in the same class; smaller than a distance between two features extracted from the same object part in the images in different classes; smaller than a distance between two features extracted from different object parts in the images in different classes;
a distance between two features extracted from different object parts in the images in the same class satisfies: smaller than a distance between two features extracted from different object parts in the images in different classes;
a distance between two features extracted from the same object part in the images in different classes satisfies: smaller than a distance between two features extracted from different object parts in the images in different classes.

### Claim 3 (depends on 1)

The method according to claim 1, wherein
the fine-grained image recognition model at least comprises two branches;
during training, regarding each image, each branch performs feature extraction for one object part in the image;
upon completion of the training, each branch respectively learns different key object parts in the image, so as to complete fine-grained image classification in combination with features extracted by each branch.

### Claim 4

A fine-grained image recognition method, wherein the method comprises:
obtaining a to-be-recognized image;
inputting the to-be-recognized image to a fine-grained image recognition model obtained by pre-training, so that the fine-grained image recognition model positions key object parts in the image, and completes fine-grained image classification according to the key object parts, and outputs a classification result.

### Claim 5 (depends on 4)

The method according to claim 4, wherein
before obtaining a to-be-recognized image, the method further comprises:
obtaining images as training samples, and respectively obtaining a tag corresponding to each image, the tag including a class to which the image belongs;
training according to the training samples and corresponding tags to obtain the fine-grained image recognition model, and performing constraint at a feature level from two dimensions, namely, the class and object parts, during the training, so that the fine-grained image recognition model learns key object parts in the images, and completes fine-grained image classification according to the key object parts.

### Claim 6 (depends on 5)

The method according to claim 5, wherein
the performing constraint at a feature level from two dimensions, namely, the class and object parts, comprises:
a distance between two features extracted from the same object part in the images in the same class simultaneously satisfies: smaller than a distance between two features extracted from different object parts in the images in the same class; smaller than a distance between two features extracted from the same object part in the images in different classes; smaller than a distance between two features extracted from different object parts in the images in different classes;
a distance between two features extracted from different object parts in the images in the same class satisfies: smaller than a distance between two features extracted from different object parts in the images in different classes;
a distance between two features extracted from the same object part in the images in different classes satisfies: smaller than a distance between two features extracted from different object parts in the images in different classes.

### Claim 7 (depends on 5)

The method according to claim 5, wherein
the fine-grained image recognition model at least comprises two branches;
during training, regarding each image, each branch performs feature extraction for one object part in the image;
upon completion of the training, each branch respectively learns different key object parts in the image, so as to complete fine-grained image classification in combination with features extracted by each branch.

### Claim 8

A computer device, comprising a memory, a processor and a computer program which is stored on the memory and runs on the processor, wherein the processor, upon executing the program, implements a method for training a fine-grained image recognition model, wherein the method comprises:
obtaining images as training samples, and respectively obtaining a tag corresponding to each image, the tag including a class to which the image belongs;
training according to the training samples and corresponding tags to obtain a fine-grained image recognition model, and performing constraint at a feature level from two dimensions, namely, the class and object parts, during the training, so that the fine-grained image recognition model learns key object parts in the images, and completes fine-grained image classification according to the key object parts.

### Claim 9 (depends on 8)

The computer device according to claim 8, wherein
the performing constraint at a feature level from two dimensions, namely, the class and object parts, comprises:
a distance between two features extracted from the same object part in the images in the same class simultaneously satisfies: smaller than a distance between two features extracted from different object parts in the images in the same class; smaller than a distance between two features extracted from the same object part in the images in different classes; smaller than a distance between two features extracted from different object parts in the images in different classes;
a distance between two features extracted from different object parts in the images in the same class satisfies: smaller than a distance between two features extracted from different object parts in the images in different classes;
a distance between two features extracted from the same object part in the images in different classes satisfies: smaller than a distance between two features extracted from different object parts in the images in different classes.

### Claim 10 (depends on 8)

The computer device according to claim 8, wherein
the fine-grained image recognition model at least comprises two branches;
during training, regarding each image, each branch performs feature extraction for one object part in the image;
upon completion of the training, each branch respectively learns different key object parts in the image, so as to complete fine-grained image classification in combination with features extracted by each branch.

### Claim 11

A computer-readable storage medium on which a computer program is stored, wherein the program, when executed by a processor, implements a method for training a fine-grained image recognition model, wherein the method comprises:
obtaining images as training samples, and respectively obtaining a tag corresponding to each image, the tag including a class to which the image belongs;
training according to the training samples and corresponding tags to obtain a fine-grained image recognition model, and performing constraint at a feature level from two dimensions, namely, the class and object parts, during the training, so that the fine-grained image recognition model learns key object parts in the images, and completes fine-grained image classification according to the key object parts.

### Claim 12 (depends on 11)

The computer-readable storage medium according to claim 11, wherein
the performing constraint at a feature level from two dimensions, namely, the class and object parts, comprises:
a distance between two features extracted from the same object part in the images in the same class simultaneously satisfies: smaller than a distance between two features extracted from different object parts in the images in the same class; smaller than a distance between two features extracted from the same object part in the images in different classes; smaller than a distance between two features extracted from different object parts in the images in different classes;
a distance between two features extracted from different object parts in the images in the same class satisfies: smaller than a distance between two features extracted from different object parts in the images in different classes;
a distance between two features extracted from the same object part in the images in different classes satisfies: smaller than a distance between two features extracted from different object parts in the images in different classes.

### Claim 13 (depends on 1)

The computer-readable storage medium according to claim 1, wherein
the fine-grained image recognition model at least comprises two branches;
during training, regarding each image, each branch performs feature extraction for one object part in the image;
upon completion of the training, each branch respectively learns different key object parts in the image, so as to complete fine-grained image classification in combination with features extracted by each branch.

### Claim 14

A computer device, comprising a memory, a processor and a computer program which is stored on the memory and runs on the processor, wherein the processor, upon executing the program, implements a fine-grained image recognition method, wherein the method comprises:
obtaining a to-be-recognized image;
inputting the to-be-recognized image to a fine-grained image recognition model obtained by pre-training, so that the fine-grained image recognition model positions key object parts in the image, and completes fine-grained image classification according to the key object parts, and outputs a classification result.

### Claim 15 (depends on 14)

The computer device according to claim 14, wherein
before obtaining a to-be-recognized image, the method further comprises:
obtaining images as training samples, and respectively obtaining a tag corresponding to each image, the tag including a class to which the image belongs;
training according to the training samples and corresponding tags to obtain the fine-grained image recognition model, and performing constraint at a feature level from two dimensions, namely, the class and object parts, during the training, so that the fine-grained image recognition model learns key object parts in the images, and completes fine-grained image classification according to the key object parts.

### Claim 16 (depends on 15)

The computer device according to claim 15, wherein
the performing constraint at a feature level from two dimensions, namely, the class and object parts, comprises:
a distance between two features extracted from the same object part in the images in the same class simultaneously satisfies: smaller than a distance between two features extracted from different object parts in the images in the same class; smaller than a distance between two features extracted from the same object part in the images in different classes; smaller than a distance between two features extracted from different object parts in the images in different classes;
a distance between two features extracted from different object parts in the images in the same class satisfies: smaller than a distance between two features extracted from different object parts in the images in different classes;
a distance between two features extracted from the same object part in the images in different classes satisfies: smaller than a distance between two features extracted from different object parts in the images in different classes.

### Claim 17 (depends on 15)

The computer device according to claim 15, wherein
the fine-grained image recognition model at least comprises two branches;
during training, regarding each image, each branch performs feature extraction for one object part in the image;
upon completion of the training, each branch respectively learns different key object parts in the image, so as to complete fine-grained image classification in combination with features extracted by each branch.

### Claim 18

A computer-readable storage medium on which a computer program is stored, wherein the program, when executed by a processor, implements a fine-grained image recognition method, wherein the method comprises:
obtaining a to-be-recognized image;
inputting the to-be-recognized image to a fine-grained image recognition model obtained by pre-training, so that the fine-grained image recognition model positions key object parts in the image, and completes fine-grained image classification according to the key object parts, and outputs a classification result.

### Claim 19 (depends on 18)

The computer-readable storage medium according to claim 18, wherein before obtaining a to-be-recognized image, the method further comprises:
obtaining images as training samples, and respectively obtaining a tag corresponding to each image, the tag including a class to which the image belongs;
training according to the training samples and corresponding tags to obtain the fine-grained image recognition model, and performing constraint at a feature level from two dimensions, namely, the class and object parts, during the training, so that the fine-grained image recognition model learns key object parts in the images, and completes fine-grained image classification according to the key object parts.

### Claim 20 (depends on 19)

The computer-readable storage medium according to claim 19, wherein
the performing constraint at a feature level from two dimensions, namely, the class and object parts, comprises:
a distance between two features extracted from the same object part in the images in the same class simultaneously satisfies: smaller than a distance between two features extracted from different object parts in the images in the same class; smaller than a distance between two features extracted from the same object part in the images in different classes; smaller than a distance between two features extracted from different object parts in the images in different classes;
a distance between two features extracted from different object parts in the images in the same class satisfies: smaller than a distance between two features extracted from different object parts in the images in different classes;
a distance between two features extracted from the same object part in the images in different classes satisfies: smaller than a distance between two features extracted from different object parts in the images in different classes.
