# US 11,907,851 B2 · Image description generation method, model training method, device and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,907,851 B2
- Document ID: US11907851B2
- Status: Granted
- Kind: B2
- Application number: 17/589,726
- Filed: 2022-01-31
- Published: 2022-05-19
- Granted: 2024-02-20
- Earliest claimed: 2017-08-30
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Wenhao Jiang; Lin Ma; Wei Liu
- Web page: https://patentgrep.com/patent/US11907851B2

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
- G06N 3/09: Supervised learning
- G06T 9/002: {using neural networks}
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/776: Validation; Performance evaluation
- G06V 10/82: using neural networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Embodiments of this application disclose an image description generation method performed at a computing device. The method includes: obtaining a target image; generating a first global feature vector and a first label vector set of the target image; generating a first multi-mode feature vector of the target image through a matching model, the matching model being a model obtained through training according to a training image and reference image description information of the training image; and applying the first multi-mode feature vector, the first global feature vector, and the first label vector set to a computing model, to obtain the target image description information, the computing model being a model obtained through training according to image description information of the training image and the reference image description information.

## Claims

### Claim 1

A computer-implemented method for generating an image description, the method comprising:
obtaining a target image;
generating a first and a second global feature vectors and a first and a second label vector sets of the target image using a feature extraction model;
generating a first and a second multi-mode feature vectors of the target image through a matching model, wherein the matching model is a model obtained through training according to a training image and reference image description information of the training image; and
applying the first and the second multi-mode feature vectors, the first and the second global feature vectors, and the first and the second label vector sets to a computing model, to obtain target image description information, wherein the computing model is a model obtained through training according to image description information of the training image and the reference image description information, further including:
splicing the first and second multi-mode feature vectors output by the matching model and the first and second label vector sets output by the feature extraction model, to obtain a first spliced vector and a second spliced vector; and
inputting the first and second spliced vectors and the first and second global feature vectors to the computing model, to obtain the target image description information.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the first multi-mode feature vector includes predicted text information of the target image.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the computing model comprises a first depth network and a second depth network;
the inputting the first and second spliced vectors and the first and second global feature vectors to the computing model, to obtain the target image description information comprises:
applying the first spliced vector and the first and second global feature vectors to the first depth network, to obtain a first output vector;
splicing the first output vector and the first and second multi-mode feature vectors, to obtain the second spliced vector; and
applying the second spliced vector to the second depth network, to obtain the target image description information.

### Claim 4 (depends on 1)

The method according to claim 1, further comprising:
obtaining a text feature vector of the reference image description information of the training image; and
training the matching model according to the second global feature vector and the text feature vector.

### Claim 5 (depends on 4)

The method according to claim 4, further comprising:
training, if the reference image description information does not match the image description information of the training image, the computing model according to the image description information and the reference image description information of the training image.

### Claim 6

A computing device, comprising one or more processors, memory coupled to the one or more processors and a plurality of programs stored in the memory that, when executed by the one or more processors, cause the computing device to perform a plurality of operations comprising:
obtaining a target image;
generating a first and a second global feature vectors and a first and a second label vector sets of the target image;
generating a first and a second multi-mode feature vectors of the target image through a matching model, wherein the matching model is a model obtained through training according to a training image and reference image description information of the training image; and
applying the first and the second multi-mode feature vectors, the first and the second global feature vectors, and the first and the second label vector sets to a computing model, to obtain target image description information, wherein the computing model is a model obtained through training according to image description information of the training image and the reference image description information, further including:
splicing the first and second multi-mode feature vectors output by the matching model and the first and second label vector sets output by the feature extraction model, to obtain a first spliced vector and a second spliced vector; and
inputting the first and second spliced vectors and the first and second global feature vectors to the computing model, to obtain the target image description information.

### Claim 7 (depends on 6)

The computing device according to claim 6, wherein the first multi-mode feature vector includes predicted text information of the target image.

### Claim 8 (depends on 6)

The computing device according to claim 6, wherein the computing model comprises a first depth network and a second depth network;
the inputting the first and second spliced vectors and the first and second global feature vectors to the computing model, to obtain the target image description information comprises:
applying the first spliced vector and the first and second global feature vectors to the first depth network, to obtain a first output vector;
splicing the first output vector and the first and second multi-mode feature vectors, to obtain the second spliced vector; and
applying the second spliced vector to the second depth network, to obtain the target image description information.

### Claim 9 (depends on 6)

The computing device according to claim 6, wherein the plurality of operations further comprise:
obtaining a text feature vector of the reference image description information of the training image; and
training the matching model according to the second global feature vector and the text feature vector.

### Claim 10 (depends on 9)

The computing device according to claim 9, wherein the plurality of operations further comprise:
training, if the reference image description information does not match the image description information of the training image, the computing model according to the image description information and the reference image description information of the training image.

### Claim 11

A non-transitory computer readable storage medium storing a plurality of machine readable instructions in connection with a computing device having one or more processors, wherein the plurality of machine readable instructions, when executed by the one or more processors, cause the computing device to perform a plurality of operations including:
obtaining a target image;
generating a first and a second global feature vectors and a first and a second label vector sets of the target image;
generating a first and a second multi-mode feature vectors of the target image through a matching model, wherein the matching model is a model obtained through training according to a training image and reference image description information of the training image; and
applying the first and the second multi-mode feature vectors, the first and the second global feature vectors, and the first and the second label vector sets to a computing model, to obtain target image description information, wherein the computing model is a model obtained through training according to image description information of the training image and the reference image description information, further including:
splicing the first and second multi-mode feature vectors output by the matching model and the first and second label vector sets output by the feature extraction model, to obtain a first spliced vector and a second spliced vector; and
inputting the first and second spliced vectors and the first and second global feature vectors to the computing model, to obtain the target image description information.

### Claim 12 (depends on 11)

The non-transitory computer readable storage medium according to claim 11, wherein the first multi-mode feature vector includes predicted text information of the target image.

### Claim 13 (depends on 11)

The non-transitory computer readable storage medium according to claim 11, wherein the computing model comprises a first depth network and a second depth network;
the inputting the first and second spliced vectors and the first and second global feature vectors to the computing model, to obtain the target image description information comprises:
applying the first spliced vector and the first and second global feature vectors to the first depth network, to obtain a first output vector;
splicing the first output vector and the first and second multi-mode feature vectors, to obtain the second spliced vector; and
applying the second spliced vector to the second depth network, to obtain the target image description information.

### Claim 14 (depends on 11)

The non-transitory computer readable storage medium according to claim 11, wherein the plurality of operations further comprise:
obtaining a text feature vector of the reference image description information of the training image; and
training the matching model according to the second global feature vector and the text feature vector.

### Claim 15 (depends on 14)

The non-transitory computer readable storage medium according to claim 14, wherein the plurality of operations further comprise:
training, if the reference image description information does not match the image description information of the training image, the computing model according to the image description information and the reference image description information of the training image.

### Claim 16 (depends on 11)

The non-transitory computer readable storage medium according to claim 11, wherein the first multi-mode feature vector includes predicted text information of the target image.

### Claim 17 (depends on 11)

The non-transitory computer readable storage medium according to claim 11, wherein the computing model comprises a first depth network and a second depth network;
the inputting the first and second spliced vectors and the first and second global feature vectors to the computing model, to obtain the target image description information comprises:
applying the first spliced vector and the first and second global feature vectors to the first depth network, to obtain a first output vector;
splicing the first output vector and the first and second multi-mode feature vectors, to obtain the second spliced vector; and
applying the second spliced vector to the second depth network, to obtain the target image description information.