# US 2022/0092351 A1 · Image classification method, neural network training method, and apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0092351 A1
- Document ID: US20220092351A1
- Status: Published application
- Kind: A1
- Application number: 17/538,640
- Filed: 2021-11-30
- Published: 2022-03-24
- Earliest claimed: 2019-05-30
- Assignees: Huawei Technologies Co., Ltd.; Peking University
- Applicants: Huawei Technologies Co., Ltd.; Peking University
- Inventors: Weiran HUANG; Aoxue LI; Zhenguo LI; Tiange LUO; Li QIAN; Liwei WANG
- Web page: https://patentgrep.com/patent/US20220092351A1

## CPC

- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/24: Classification techniques
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/40: Extraction of image or video features
- G06V 10/82: using neural networks
- G06V 10/95: {structured as a network, e.g. client-server architectures}
- G06N 3/048: Activation functions
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An image classification method, a neural network training method, and an apparatus are provided, and relate to the field of artificial intelligence, and specifically, to the field of computer vision. The image classification method includes: obtaining a to-be-processed image; and obtaining a classification result of the to-be-processed image based on a pre-trained neural network model, where the classification result includes a class or a superclass to which the to-be-processed image belongs. When the neural network model is trained, not only labels of a plurality of training images but also class hierarchy information of the plurality of training images is used. That is, more abundant information of the training images is used. Therefore, images can be better classified.

## Claims

### Claim 1

An image classification method, comprising:
obtaining a to-be-processed image; and
classifying the to-be-processed image based on a preset neural network model, to obtain a classification result of the to-be-processed image,
wherein the classification result comprises a class or a superclass to which the to-be-processed image belongs,
wherein the neural network model is obtained by performing training based on a plurality of training images, a label of each training image of the plurality of training images, and class hierarchy information of the plurality of training images, and
wherein the class hierarchy information of the plurality of training images comprises at least one superclass to which each of a plurality of classes belongs, and the plurality of classes comprise one or more classes to which the plurality of training images belong.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the at least one superclass to which each of the plurality of classes belongs is determined based on association degree information between word vectors corresponding to the labels of the plurality of training images.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the at least one superclass to which each of the plurality of classes belongs is pre-labeled.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the classifying the to-be-processed image based on the preset neural network model comprises:
extracting a feature vector of the to-be-processed image;
determining, based on the feature vector of the to-be-processed image, confidence that the to-be-processed image belongs to each of a plurality of candidate classes; and
determining the classification result of the to-be-processed image from the plurality of candidate classes based on the confidence that the to-be-processed image belongs to each of the plurality of candidate classes.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the determining the classification result of the to-be-processed image from the plurality of candidate classes based on the confidence that the to-be-processed image belongs to each of the plurality of candidate classes comprises:
determining a first candidate class in the plurality of candidate classes as the classification result of the to-be-processed image, wherein the first candidate class is a class with highest confidence in the plurality of candidate classes.

### Claim 6 (depends on 5)

The method according to claim 5, wherein the method further comprises:
determining a first candidate superclass in a plurality of candidate superclasses as the classification result of the to-be-processed image, wherein the confidence of the first candidate class is less than a first confidence threshold, and confidence of the first candidate superclass is greater than or equal to a second confidence threshold.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the classifying the to-be-processed image based on the preset neural network model comprises:
obtaining reference images of the to-be-processed image, wherein the reference images comprise a plurality of image classes, and the to-be-processed image belongs to one of the plurality of image classes;
extracting a feature vector of the to-be-processed image and a feature vector of each image class in the plurality of image classes;
determining, based on a difference between the feature vector of the to-be-processed image and the feature vector of each image class in the plurality of image classes, of confidence that the to-be-processed image belongs to each image class in the plurality of image classes; and
determining the classification result of the to-be-processed image from the plurality of image classes based on the confidence that the to-be-processed image belongs to each image class in the plurality of image classes.

### Claim 8

An image classification method, comprising:
obtaining a to-be-processed image; and
classifying the to-be-processed image based on a preset first neural network model, to obtain a classification result of the to-be-processed image,
wherein the classification result of the to-be-processed image comprises a class or a superclass to which the to-be-processed image belongs,
wherein the first neural network model is obtained by performing training based on a plurality of first feature vectors, labels of a plurality of first training images, and semantic description information of the plurality of first training images, wherein the semantic description information of each first training image of the plurality of first training images is a semantic description of an image feature of the first training image; and
wherein the plurality of first feature vectors are feature vectors obtained by performing feature extraction on the plurality of first training images by a second neural network model, wherein the second neural network model is obtained by performing training based on a plurality of second training images, a label of each of the plurality of second training images, and class hierarchy information of the plurality of second training images, and wherein the class hierarchy information of the plurality of second training images comprises one or more classes to which the plurality of second training images belong and at least one superclass to which each of the plurality of second training images belongs.

### Claim 9 (depends on 8)

The method according to claim 8, wherein the classifying the to-be-processed image based on the preset first neural network model comprises:
extracting a feature vector of the to-be-processed image based on the second neural network model;
processing the feature vector of the to-be-processed image based on the first neural network model, to obtain a semantic vector of the to-be-processed image; and
comparing the semantic vector of the to-be-processed image with a candidate semantic vector, to obtain the classification result of the to-be-processed image.

### Claim 10

A neural network training method, comprising:
obtaining a plurality of training images;
extracting image features of the plurality of training images based on a feature extraction network of a neural network;
processing the image features of the plurality of training images based on a hierarchical prediction network of the neural network, to obtain classification results of the plurality of training images, wherein a classification result of each training image of the plurality of training images comprises a class and a superclass to which the training image belongs; and
determining a parameter of a neural network model based on the classification results of the plurality of training images and labeled classes of the plurality of training images.

### Claim 11

An image classification apparatus, comprising:
a memory configured to store a program; and
a processor configured to execute the program stored in the memory, wherein when the program stored in the memory is executed by the processor, the processor is configured to:
obtain a to-be-processed image; and
classify the to-be-processed image based on a preset neural network model, to obtain a classification result of the to-be-processed image,
wherein the classification result comprises a class or a superclass to which the to-be-processed image belongs,
wherein the neural network model is obtained by performing training based on a plurality of training images, a label of each training image of the plurality of training images, and class hierarchy information of the plurality of training images, and
wherein the class hierarchy information of the plurality of training images comprises at least one superclass to which each of a plurality of classes belongs, and the plurality of classes comprise one or more classes to which the plurality of training images belong.

### Claim 12 (depends on 11)

The apparatus according to claim 11, wherein the at least one superclass to which each of the plurality of classes belongs is determined based on association degree information between word vectors corresponding to the labels of the plurality of training images.

### Claim 13 (depends on 11)

The apparatus according to claim 11, wherein the at least one superclass to which each of the plurality of classes belongs is pre-labeled.

### Claim 14 (depends on 11)

The apparatus according to claim 11, wherein the classifying the to-be-processed image based on the preset neural network model comprises:
extracting a feature vector of the to-be-processed image;
determining, based on the feature vector of the to-be-processed image, confidence that the to-be-processed image belongs to each of a plurality of candidate classes; and
determining the classification result of the to-be-processed image from the plurality of candidate classes based on the confidence that the to-be-processed image belongs to each of the plurality of candidate classes.

### Claim 15 (depends on 14)

The apparatus according to claim 14, wherein the determining the classification result of the to-be-processed image from the plurality of candidate classes based on the confidence that the to-be-processed image belongs to each of the plurality of candidate classes comprises:
determining a first candidate class in the plurality of candidate classes as the classification result of the to-be-processed image, wherein the first candidate class is a class with highest confidence in the plurality of candidate classes.

### Claim 16 (depends on 15)

The apparatus according to claim 15, wherein the processor is further configured to:
determine a first candidate superclass in a plurality of candidate superclasses as the classification result of the to-be-processed image, wherein the confidence of the first candidate class is less than a first confidence threshold, and confidence of the first candidate superclass is greater than or equal to a second confidence threshold.

### Claim 17 (depends on 11)

The apparatus according to claim 11, wherein the classifying the to-be-processed image based on the preset neural network model comprises:
obtaining reference images of the to-be-processed image, wherein the reference images comprise a plurality of image classes, and the to-be-processed image belongs to one of the plurality of image classes;
extracting a feature vector of the to-be-processed image and a feature vector of each image class in the plurality of image classes;
determining, based on a difference between the feature vector of the to-be-processed image and the feature vector of each image class in the plurality of image classes, confidence that the to-be-processed image belongs to each image class in the plurality of image classes; and
determining the classification result of the to-be-processed image from the plurality of image classes based on the confidence that the to-be-processed image belongs to each image class in the plurality of image classes.

### Claim 18

An image classification apparatus, comprising:
a memory configured to store a program; and
a processor configured to execute the program stored in the memory, wherein when the program stored in the memory is executed by the processor, the processor is configured to:
obtain a to-be-processed image; and
classify the to-be-processed image based on a preset first neural network model, to obtain a classification result of the to-be-processed image,
wherein the classification result of the to-be-processed image comprises a class or a superclass to which the to-be-processed image belongs,
wherein the first neural network model is obtained by performing training based on a plurality of first feature vectors, labels of a plurality of first training images, and semantic description information of the plurality of first training images, wherein the semantic description information of each first training image of the plurality of first training images is a semantic description of an image feature of the first training image; and
wherein the plurality of first feature vectors are feature vectors obtained by performing feature extraction on the plurality of first training images by a second neural network model, wherein the second neural network model is obtained by performing training based on a plurality of second training images, a label of each of the plurality of second training images, and class hierarchy information of the plurality of second training images, and wherein the class hierarchy information of the plurality of second training images comprises one or more classes to which the plurality of second training images belong and at least one superclass to which each of the plurality of second training images belongs.

### Claim 19 (depends on 18)

The apparatus according to claim 18, wherein the classifying the to-be-processed image based on the preset first neural network model comprises:
extracting a feature vector of the to-be-processed image based on the second neural network model;
processing the feature vector of the to-be-processed image based on the first neural network model, to obtain a semantic vector of the to-be-processed image; and
comparing the semantic vector of the to-be-processed image with a candidate semantic vector, to obtain the classification result of the to-be-processed image.

### Claim 20

A neural network training apparatus, comprising:
a memory configured to store a program; and
a processor configured to execute the program stored in the memory, wherein when the program stored in the memory is executed by the processor, the processor is configured to:
obtain a plurality of training images;
extract image features of the plurality of training images based on a feature extraction network of a neural network;
process the image features of the plurality of training images based on a hierarchical prediction network of the neural network, to obtain classification results of the plurality of training images, wherein a classification result of each training image of the plurality of training images comprises a class and a superclass to which the training image belongs; and
determine a parameter of a neural network model based on the classification results of the plurality of training images and labeled classes of the plurality of training images.

### Claim 21 (depends on 1)

A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable medium stores program code to be executed by a device, and the program code is used to perform the method according to claim 1.

### Claim 22 (depends on 1)

A chip, wherein the chip comprises a processor and a data interface, and the processor reads, through the data interface, instructions stored in a memory, to perform the method according to claim 1.
