# US 10,504,010 B2 · Systems and methods for fast novel visual concept learning from sentence descriptions of images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,504,010 B2
- Document ID: US10504010B2
- Status: Granted
- Kind: B2
- Application number: 15/418,401
- Filed: 2017-01-27
- Published: 2017-05-25
- Granted: 2019-12-10
- Earliest claimed: 2015-10-02
- Assignees: Baidu USA LLC
- Applicants: Baidu USA, LLC
- Inventors: Junhua Mao; Wei Xu; Yi Yang; Jiang Wang; Zhiheng Huang
- Web page: https://patentgrep.com/patent/US10504010B2

## CPC

- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06F 18/2413: based on distances to training or reference patterns
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/096: Transfer learning
- G06V 30/274: {Syntactic or semantic context, e.g. balancing}
- G06F 40/30: Semantic analysis
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Described herein are systems and methods that address the task of learning novel visual concepts, and their interactions with other concepts, from a few images with sentence descriptions. Using linguistic context and visual features, embodiments are able to efficiently hypothesize the semantic meaning of new words and add them to model word dictionaries so that they can be used to describe images which contain these novel concepts. In the experiments, it was shown that the tested embodiments effectively learned novel visual concepts from a few examples without disturbing the previously learned concepts.

## Claims

### Claim 1

A computer-implemented method for enlarging a trained model's word dictionary to describe a novel concept without requiring a complete retraining of the trained model, the method comprising:
inputting into a multimodal recurrent neural network a set of images and associated captions related to one or more novel concepts not trained in the trained model, the multimodal recurrent neural network comprising:
a convolution neural network layer component that generates an image representation of an input image from the set of images;
at least one word embedding component that encodes meaning of a word from a caption associated with the input image into a word representation;
a recurrent layer component that maps a recurrent layer activation of a prior time frame into a same vector space as a word representation at a current time frame and combines them;
a multimodal component that is distinct from the recurrent layer component and receives a first input from the recurrent layer component and a second input from the convolution neural network layer component and combines them; and
a SoftMax layer component that uses an output of the multimodal component to generate a probability distribution of a next word in a caption-level description;
fixing at least some of a set of originally learned weights of the trained model during training of the multimodal recurrent neural network using the set of images and associated captions related to one or more novel concepts; and
fixing at least some of a set of baseline probabilities of the trained model during training of the multimodal recurrent neural network using the set of images and associated captions related to one or more novel concepts.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1 wherein the multimodal component further receives a third input from the at least one word embedding component.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2 wherein the at least one word embedding layer comprises two word embedding layers, in which an output of a first word embedding layer is provided as an input to a second word embedding layer and an output of the second word embedding layer is provided as the third input to the multimodal component.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1 wherein the recurrent layer component comprises a Long-Short Term Memory (LSTM) layer.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1 wherein the at least one word embedding component receives a weight matrix, UD, from a one-hot layer that represents a word from a caption as an index in the word dictionary.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5 further comprising:
decomposing a second weight matrix between the multimodal component and the SoftMax layer component that is used to compute activation of a SoftMax layer in the SoftMax layer component into two sub-matrices; and
sharing, in a transposed manner, at least some of values from the weight matrix between the one-hot layer and at least one word embedding layer with at least some values of one of the sub-matrices.

### Claim 7 (depends on 5)

The computer-implemented method of claim 5 wherein the step of fixing at least some of a set of originally learned weights of the trained model during training of the multimodal recurrent neural network using the set of images and associated captions related to one or more novel concepts comprises:
separating the weight matrix, UD, between the one-hot layer and at least one word embedding layer into two sub-matrices UD=[UDo; UDn], where UDo and UDn are associated with original words in the work dictionary and the new words of the one or more novel concepts, respectively;
fixing the sub-matrix UDo during training of the one or more novel concepts; and
updating the sub-matrix UDn.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1 wherein the step of fixing at least some of a set of baseline probabilities of the trained model during training of the multimodal recurrent neural network using the set of images and associated captions related to one or more novel concepts comprises:
separating a set of baseline probabilities, b, used in computing activation of a SoftMax layer in the SoftMax layer component into two parts: b=[bo; bn], where bo and bn are associated with original words in the work dictionary and new words of the one or more novel concepts, respectively;
converting the original set of baseline probabilities, b, into b′ using a centralized activation of an intermediate layer formed between the multimodal component and the SoftMax layer component;
setting every element in b′n to be an average value of element in b′o; and
fixing b′n during training of the one or more novel concepts.

### Claim 9

A computer-implemented method for supplemental training of a trained model to describe a novel concept without requiring a complete retraining of the trained model, the method comprising:
inputting into the trained model a set of images and associated captions related to a novel concept not trained in the trained model, the trained model comprising:
a language component comprising at least one word embedding layer and a recurrent layer that maps an index of a word in a word dictionary and taken from a caption into a semantic dense word embedding space and stores word context information in the recurrent layer;
a vision component that generates an image representation of an input image from the set of images;
a multimodal component, which is distinct from the recurrent layer of the language component, that generates a representation using information from the language component and the vision component; and
a SoftMax layer component that receives input obtained from the multimodal component and predicts an index of a next word;
fixing at least some of a set of originally learned weights of the trained model during the supplemental training of the trained model; and
fixing at least some of a set of baseline probabilities of the trained model during the supplemental training of the trained model.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9 wherein the multimodal component further receives a third input from the at least one word embedding component.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10 wherein the at least one word embedding layer comprises two word embedding layers, in which an output of a first word embedding layer is provided as an input to a second word embedding layer and an output of the second word embedding layer is provided as the third input to the multimodal component.

### Claim 12 (depends on 9)

The computer-implemented method of claim 9 wherein the recurrent layer comprises a Long-Short Term Memory (LSTM) layer.

### Claim 13 (depends on 9)

The computer-implemented method of claim 9 wherein the at least one word embedding layer receives a weight matrix, UD, from a one-hot layer that represents a word from a caption as an index in the word dictionary, and the computer-implemented method further comprises:
decomposing a second weight matrix between the multimodal component and the SoftMax layer component that is used to compute activation of a SoftMax layer in the SoftMax layer component into two sub-matrices; and
sharing, in a transposed manner, at least some of values from the weight matrix between the one-hot layer and at least one word embedding layer with at least some values of one of the sub-matrices.

### Claim 14 (depends on 13)

The computer-implemented method of claim 13 wherein the at least one word embedding layer receives a weight matrix, UD, from a one-hot layer that represents a word from a caption as an index in the word dictionary and wherein the step of fixing at least some of a set of originally learned weights of the trained model during the supplemental training of the trained model comprises:
separating the weight matrix, UD, between the one-hot layer and at least one word embedding layer into two sub-matrices UD=[UDo; UDn], where UDo and UDn are associated with original words in the work dictionary and the new words of the one or more novel concepts, respectively;
fixing the sub-matrix UDo during the supplemental training; and
updating the submatrix UDn.

### Claim 15 (depends on 9)

The computer-implemented method of claim 9 wherein the step of fixing at least some of a set of baseline probabilities of the trained model during the supplemental training of the trained model comprises:
separating a set of baseline probabilities, b, used in computing activation of a SoftMax layer in the SoftMax layer component into two parts: b=[bo; bn], where bo and bn are associated with original words in the work dictionary and new words of the one or more novel concepts, respectively;
converting the original set of baseline probabilities, b, into b′ using a centralized activation of an intermediate layer formed between the multimodal component and the SoftMax layer component; and
setting every element in b′n to be an average value of element in b′o; and
fixing b′n during training of the one or more novel concepts.

### Claim 16

A system for providing supplemental training of a trained model to describe a novel concept without requiring a complete retraining of the trained model, the system comprising:
an input interface for receiving a set of images and associated captions related to a novel concept not trained in the trained model;
one or more processors; and
a non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by the one or more processors, causes steps to be performed comprising:
inputting into the trained model a set of images and associated captions related to a novel concept not trained in the trained model, the trained model comprising:
a language component comprising at least one word embedding layer and a recurrent layer that maps an index of a word in a word dictionary and taken from a caption into a semantic dense word embedding space and stores word context information in the recurrent layer;
a vision component that generates an image representation of an input image from the set of images;
a multimodal component, which is distinct from the recurrent layer of the language component, that generates a representation using information from the language component and the vision component; and
a SoftMax layer component that receives input obtained from the multimodal component and predicts an index of a next word;
fixing at least some of a set of originally learned weights of the trained model during the supplemental training of the trained model; and
fixing at least some of a set of baseline probabilities of the trained model during the supplemental training of the trained model.

### Claim 17 (depends on 16)

The system of claim 16 wherein the multimodal component further receives a third input from the at least one word embedding component.

### Claim 18 (depends on 17)

The system of claim 17 wherein the at least one word embedding layer comprises two word embedding layers, in which an output of a first word embedding layer is provided as an input to a second word embedding layer and an output of the second word embedding layer is provided as the third input to the multimodal component.

### Claim 19 (depends on 16)

The system of claim 16 wherein the recurrent layer comprises a Long-Short Term Memory (LSTM) layer.

### Claim 20 (depends on 16)

The system of claim 16 wherein the at least one word embedding layer receives a weight matrix, UD, from a one-hot layer that represents a word from a caption as an index in the word dictionary, and the non-transitory computer-readable medium or media further comprises one or more sequences of instructions which, when executed by the one or more processors, causes steps to be performed comprising:
decomposing a second weight matrix between the multimodal component and the SoftMax layer component that is used to compute activation of a SoftMax layer in the SoftMax layer component into two sub-matrices; and
sharing, in a transposed manner, at least some of values from the weight matrix between the one-hot layer and at least one word embedding layer with at least some values of one of the sub-matrices.