# US 2019/0377979 A1 · Image description generation method, model training method, device and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0377979 A1
- Document ID: US20190377979A1
- Status: Published application
- Kind: A1
- Application number: 16/548,621
- Filed: 2019-08-22
- Published: 2019-12-12
- Earliest claimed: 2017-08-30
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Wenhao JIANG; Lin MA; Wei LIU
- Web page: https://patentgrep.com/patent/US20190377979A1

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
- G06N 3/09: Supervised learning
- G06T 9/002: {using neural networks}
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/776: Validation; Performance evaluation
- G06V 10/82: using neural networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Embodiments of this application disclose an image description generation method performed at a computing device. The method includes: obtaining a target image; generating a first global feature vector and a first label vector set of the target image; applying the target image to a matching model and generating a first multi-mode feature vector of the target image through the matching model, the matching model being a model obtained through training according to a training image and reference image description information of the training image; and generating target image description information of the target image according to the first multi-mode feature vector, the first global feature vector, and the first label vector set.

## Claims

### Claim 1

An image description generation method, applied to a computing device having one or more processors and memory storing programs to be executed by the one or more processors, the method comprising:
obtaining a target image;
generating a first global feature vector and a first label vector set of the target image;
applying the target image to a matching model and generating a first multi-mode feature vector of the target image through the matching model, the matching model being a model obtained through training according to a training image and reference image description information of the training image; and
generating target image description information of the target image according to the first multi-mode feature vector, the first global feature vector, and the first label vector set.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the generating target image description information of the target image according to the first multi-mode feature vector, the first global feature vector, and the first label vector set comprises:
applying the first multi-mode feature vector, the first global feature vector, and the first label vector set to a computing model, to obtain the target image description information, wherein the computing model is a model obtained through training according to image description information and the reference image description information of the training image.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the computing model comprises n depth networks, and n is a positive integer;
the applying the first multi-mode feature vector, the first global feature vector, and the first label vector set to a computing model, to obtain the target image description information comprises:
generating the target image description information according to the first multi-mode feature vector, the first global feature vector, the first label vector set, and the n depth networks;
wherein an input parameter of at least one depth network in the n depth networks comprises a spliced vector, when an input parameter of the ith depth network comprises the spliced vector, if i is equal to 1, the spliced vector is a vector obtained by splicing the first multi-mode feature vector and the first label vector set, if i is greater than 1, the spliced vector is a vector obtained by splicing an output vector of the (i−1)th depth network and the first multi-mode feature vector, wherein i is greater than or equal to 1, and less than or equal to n.

### Claim 4 (depends on 3)

The method according to claim 3, wherein n is equal to 2;
the generating the target image description information according to the first multi-mode feature vector, the first global feature vector, the first label vector set, and the n depth networks comprises:
splicing the first multi-mode feature vector and the first label vector set, to obtain a first spliced vector;
applying the first spliced vector and the first global feature vector to a first depth network, to obtain a first output vector;
splicing the first output vector and the first multi-mode feature vector, to obtain a second spliced vector; and
applying the second spliced vector to a second depth network, to obtain the target image description information.

### Claim 5 (depends on 1)

The method according to claim 1, further comprising:
obtaining a second global feature vector and a second label vector set of the training image, and a text feature vector of the reference image description information of the training image; and
training the matching model according to the second global feature vector and the text feature vector.

### Claim 6 (depends on 5)

The method according to claim 5, further comprising:
generating a second multi-mode feature vector of the training image through the matching model obtained through training;
applying the second multi-mode feature vector, the second global feature vector, and the second label vector set to the computing model, to obtain the image description information of the training image; and
training, if the reference image description information does not match the image description information of the training image, the computing model according to the image description information and the reference image description information of the training image.

### Claim 7

A computing device, comprising one or more processors, memory coupled to the one or more processors and a plurality of programs stored in the memory that, when executed by the one or more processors, cause the computing device to perform a plurality of operations comprising:
obtaining a target image;
generating a first global feature vector and a first label vector set of the target image;
applying the target image to a matching model and generating a first multi-mode feature vector of the target image through the matching model, the matching model being a model obtained through training according to a training image and reference image description information of the training image; and
generating target image description information of the target image according to the first multi-mode feature vector, the first global feature vector, and the first label vector set.

### Claim 8 (depends on 7)

The computing device according to claim 7, wherein the generating target image description information of the target image according to the first multi-mode feature vector, the first global feature vector, and the first label vector set comprises:
applying the first multi-mode feature vector, the first global feature vector, and the first label vector set to a computing model, to obtain the target image description information, wherein the computing model is a model obtained through training according to image description information and the reference image description information of the training image.

### Claim 9 (depends on 8)

The computing device according to claim 8, wherein the computing model comprises n depth networks, and n is a positive integer;
the applying the first multi-mode feature vector, the first global feature vector, and the first label vector set to a computing model, to obtain the target image description information comprises:
generating the target image description information according to the first multi-mode feature vector, the first global feature vector, the first label vector set, and the n depth networks;
wherein an input parameter of at least one depth network in the n depth networks comprises a spliced vector, when an input parameter of the ith depth network comprises the spliced vector, if i is equal to 1, the spliced vector is a vector obtained by splicing the first multi-mode feature vector and the first label vector set, if i is greater than 1, the spliced vector is a vector obtained by splicing an output vector of the (i−1)th depth network and the first multi-mode feature vector, wherein i is greater than or equal to 1, and less than or equal to n.

### Claim 10 (depends on 9)

The computing device according to claim 9, wherein n is equal to 2;
the generating the target image description information according to the first multi-mode feature vector, the first global feature vector, the first label vector set, and the n depth networks comprises:
splicing the first multi-mode feature vector and the first label vector set, to obtain a first spliced vector;
applying the first spliced vector and the first global feature vector to a first depth network, to obtain a first output vector;
splicing the first output vector and the first multi-mode feature vector, to obtain a second spliced vector; and
applying the second spliced vector to a second depth network, to obtain the target image description information.

### Claim 11 (depends on 7)

The computing device according to claim 7, wherein the plurality of operations further comprise:
obtaining a second global feature vector and a second label vector set of the training image, and a text feature vector of the reference image description information of the training image; and
training the matching model according to the second global feature vector and the text feature vector.

### Claim 12 (depends on 11)

The computing device according to claim 11, wherein the plurality of operations further comprise:
generating a second multi-mode feature vector of the training image through the matching model obtained through training;
applying the second multi-mode feature vector, the second global feature vector, and the second label vector set to the computing model, to obtain the image description information of the training image; and
training, if the reference image description information does not match the image description information of the training image, the computing model according to the image description information and the reference image description information of the training image.

### Claim 13

A non-transitory computer readable storage medium storing a plurality of machine readable instructions in connection with a computing device having one or more processors, wherein the plurality of machine readable instructions, when executed by the one or more processors, cause the computing device to perform a plurality of operations including:
obtaining a target image;
generating a first global feature vector and a first label vector set of the target image;
applying the target image to a matching model and generating a first multi-mode feature vector of the target image through the matching model, the matching model being a model obtained through training according to a training image and reference image description information of the training image; and
generating target image description information of the target image according to the first multi-mode feature vector, the first global feature vector, and the first label vector set.

### Claim 14 (depends on 13)

The non-transitory computer readable storage medium according to claim 13, wherein the generating target image description information of the target image according to the first multi-mode feature vector, the first global feature vector, and the first label vector set comprises:
applying the first multi-mode feature vector, the first global feature vector, and the first label vector set to a computing model, to obtain the target image description information, wherein the computing model is a model obtained through training according to image description information and the reference image description information of the training image.

### Claim 15 (depends on 14)

The non-transitory computer readable storage medium according to claim 14, wherein the computing model comprises n depth networks, and n is a positive integer;
the applying the first multi-mode feature vector, the first global feature vector, and the first label vector set to a computing model, to obtain the target image description information comprises:
generating the target image description information according to the first multi-mode feature vector, the first global feature vector, the first label vector set, and the n depth networks;
wherein an input parameter of at least one depth network in the n depth networks comprises a spliced vector, when an input parameter of the ith depth network comprises the spliced vector, if i is equal to 1, the spliced vector is a vector obtained by splicing the first multi-mode feature vector and the first label vector set, if i is greater than 1, the spliced vector is a vector obtained by splicing an output vector of the (i−1)th depth network and the first multi-mode feature vector, wherein i is greater than or equal to 1, and less than or equal to n.

### Claim 16 (depends on 15)

The non-transitory computer readable storage medium according to claim 15, wherein n is equal to 2;
the generating the target image description information according to the first multi-mode feature vector, the first global feature vector, the first label vector set, and the n depth networks comprises:
splicing the first multi-mode feature vector and the first label vector set, to obtain a first spliced vector;
applying the first spliced vector and the first global feature vector to a first depth network, to obtain a first output vector;
splicing the first output vector and the first multi-mode feature vector, to obtain a second spliced vector; and
applying the second spliced vector to a second depth network, to obtain the target image description information.

### Claim 17 (depends on 13)

The non-transitory computer readable storage medium according to claim 13, wherein the plurality of operations further comprise:
obtaining a second global feature vector and a second label vector set of the training image, and a text feature vector of the reference image description information of the training image; and
training the matching model according to the second global feature vector and the text feature vector.

### Claim 18 (depends on 17)

The non-transitory computer readable storage medium according to claim 17, wherein the plurality of operations further comprise:
generating a second multi-mode feature vector of the training image through the matching model obtained through training;
applying the second multi-mode feature vector, the second global feature vector, and the second label vector set to the computing model, to obtain the image description information of the training image; and
training, if the reference image description information does not match the image description information of the training image, the computing model according to the image description information and the reference image description information of the training image.