Back to results
US 12,755,099 B2Granted
Method and apparatus for training named entity recognition model and non-transitory computer-readable medium
AssigneeRicoh Company, Ltd.
InventorsYuming Zhang, Bin Dong, Shanshan Jiang, Yongwei Zhang
Life of this patent
Estimating the term…
Abstract
A method and an apparatus are provided for training a named entity recognition (NER) model. By constructing tag annotations for tags and causing the tag annotations to contain information for indicating the positions of tokens in named entities, corresponding to the tags, respectively, in the process of training the NER model, the NER model can better understand the different positions of different tokens in the same named entity, so that the trained NER model can more accurately recognize named entities.
Claims15
A method of training a named entity recognition model, wherein, the named entity recognition model includes an encoder and a decoder, and the encoder contains a pre-trained language model and an attention mechanism model,
- the method comprising:
- acquiring a plurality of training texts, wherein, each training text is pre-marked with tags, and the tags are used to mark named entity types to which tokens in the training text belong, and constructing a tag annotation for each of the tags, wherein, in response to each of the tags being a tag corresponding to a named entity, the tag annotation includes a position indication token indicating a position of the token in the named entity, corresponding to the tag;
- generating a weight matrix on the basis of all the tag annotations, wherein, each row of the weight matrix corresponds to one tag annotation, respective elements in the row sequentially correspond to the tokens in the tag annotation, values of the elements corresponding to the position indication tokens in the tag annotation are k, values of the elements corresponding to the tokens other than the position indication tokens in the tag annotation are 0, and k is a learnable parameter during a process of training the named entity recognition model, wherein the generation of the weight matrix includes:
- unifying numbers of the tokens of all the tag annotations on the basis of a maximum number of tokens in all the tag annotations;
- initializing a zero matrix, wherein, each row of the zero matrix corresponds to one tag annotation, and respective elements in each row sequentially correspond to the tokens in the tag annotation;
- and
- setting values of the elements in the zero matrix, corresponding to the position indication tokens in all the tag annotations to k, so as to obtain the weight matrix, wherein, an initial value of k is 1;
- inputting the training text and the tag annotations into the pre-trained language model to obtain a first vector representation of the training text and a first vector representation of the tag annotations;
- inputting the first vector representation of the training text and the first vector representation of the tag annotations into the attention mechanism model to calculate a first relationship between the training text and the tag annotations, weighting the first relationship by using the weight matrix to obtain a second relationship, and generating a final vector representation of the training text on the basis of the second relationship;
- inputting the final vector representation of the training text into the decoder to obtain a tag corresponding to each token in the training text, output by the decoder;
- and
- optimizing the named entity recognition model on the basis of the tag corresponding to each token in the training text, output by the decoder and the pre-marked tags in the training text to obtain a trained named entity recognition model.
9 more elements
An apparatus for training a named entity recognition model, wherein, the named entity recognition model includes an encoder and a decoder, and the encoder contains a pre-trained language model and an attention mechanism model,
- the apparatus comprising:
- a first acquisition part configured to acquire a plurality of training texts, wherein, each training text is pre-marked with tags, and the tags are used to mark named entity types to which tokens in the training text belong, and construct a tag annotation for each of the tags, wherein, in response to each of the tags being a tag corresponding to a named entity, the tag annotation includes a position indication token indicating a position of the token in the named entity, corresponding to the tag;
- a first generation part configured to generate a weight matrix on the basis of all the tag annotations, wherein, each row of the weight matrix corresponds to one tag annotation, respective elements in the row sequentially correspond to the tokens in the tag annotation, values of the elements corresponding to the position indication tokens in the tag annotation are k, values of the elements corresponding to the tokens other than the position indication tokens in the tag annotation are 0, and k is a learnable parameter during a process of training the named entity recognition model, wherein the first generation part is further configured to:
- unify numbers of the tokens of all the tag annotations on the basis of a maximum number of tokens in all the tag annotations;
- initialize a zero matrix, wherein, each row of the zero matrix corresponds to one tag annotation, and respective elements in each row sequentially correspond to the tokens in the tag annotation;
- and
- set values of the elements in the zero matrix, corresponding to the position indication tokens in all the tag annotations to k, so as to obtain the weight matrix, wherein, an initial value of k is 1;
- a first obtainment part configured to input the training text and the tag annotations into the pre-trained language model to obtain a first vector representation of the training text and a first vector representation of the tag annotations;
- a second obtainment part configured to input the first vector representation of the training text and the first vector representation of the tag annotations into the attention mechanism model to calculate a first relationship between the training text and the tag annotations, weight the first relationship by using the weight matrix to obtain a second relationship, and generate a final vector representation of the training text on the basis of the second relationship;
- a third obtainment part configured to input the final vector representation of the training text into the decoder to obtain a tag corresponding to each token in the training text, output by the decoder;
- and
- an optimization part configured to optimize the named entity recognition model on the basis of the tag corresponding to each token in the training text, output by the decoder and the pre-marked tags in the training text to obtain a trained named entity recognition model.
9 more elements
Dependencies are inferred from the claim text.