# US 2021/0240929 A1 · Systems and methods for end-to-end deep reinforcement learning based coreference resolution

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0240929 A1
- Document ID: US20210240929A1
- Status: Published application
- Kind: A1
- Application number: 16/960,014
- Filed: 2019-07-25
- Published: 2021-08-05
- Earliest claimed: 2019-07-25
- Assignees: Baidu USA LLC; Baidu.Com Times Technology (Beijing) Co., Ltd.
- Applicants: Baidu USA, LLC; Baidu.com Times Technology (Beijing) Co., Ltd.
- Inventors: Hongliang FEI; Xu LI; Dingcheng LI; Ping LI
- Web page: https://patentgrep.com/patent/US20210240929A1

## CPC

- G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30)
- G06F 40/279: Recognition of textual entities
- G06F 40/30: Semantic analysis
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Described herein are embodiments for end-to-end reinforcement learning based coreference resolution models to directly optimize coreference evaluation metrics. Embodiments of a reinforced policy gradient model are disclosed to incorporate reward associated with a sequence of coreference linking actions. Furthermore, maximum entropy regularization may be used for adequate exploration to prevent a model embodiment from prematurely converging to a bad local optimum. Experiments on datasets compared with state-of-the-art methods verified the effectiveness of embodiments.

## Claims

### Claim 1

A computer-implement method for training a coreference resolution model comprising:
[a] inputting a document comprising a set of text into a policy network to identify mentions in the document;
[b] given a current identified mention in the document, using the policy network to obtain a probability distribution of a set of actions in which the set of actions comprise linking the current identified mention with a prior identified mention or not linking the current identified mention to any prior identified mention;
[c] selecting an action from the set of actions using the probability distribution of actions;
[d] based upon the selected action, updating a coreference graph for the document, in which the coreference graph comprises mentions as nodes and links representing coreference connections between mentions;
[e] responsive to the document having another mention, selecting it as the current identified mention and returning to step [b];
[f] responsive to the document not having another mention, outputting the coreference graph for the document;
[g] using the outputted coreference graph and ground truth coreference information for the document, computing a reward based upon one or more metrics;
[h] using a trajectory of actions selected and the reward to compute a gradient; and
[i] updating the policy network using the gradient.

### Claim 2 (depends on 1)

The computer-implement method of claim 1 wherein the policy network is pre-trained using training steps comprising:
inputting a set of documents into the policy network that identifies mentions in the documents and generates a coreference graph for each document;
using corresponding ground-truth coreference graphs for the document to compute a loss relative to the generated coreference graphs obtained from the policy network;
using the loss to update the policy network; and
iterating the above training steps until a stopping condition is reached, the step condition comprises one or more criteria from number of epochs, error level, or number of iterations.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1 further comprising:
repeating the steps of [a]-[f] for the document to obtain a set of coreference graphs and a corresponding set of trajectories of actions for each document in an iterative operation;
obtaining a sample set of coreference graphs from the set of coreference graphs;
computing a reward for each coreference graph from the sample set of coreference graphs; and
using the rewards and trajectories of actions in the sample set to compute a gradient.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1 wherein inputting a document comprising a set of text into a policy network to identify mentions in the document comprises:
generating, using a character and word embeddings encoder, a plurality of embeddings with each embedding as a concatenation of fixed pretrained word embeddings and convolutional neural network (CNN) character embeddings;
computing and concatenating, using a bidirectional Long short-term memory (LSTM) layer, contextualized representation of each word in the input document from two directions;
performing iterative operations comprising:
generating, with head-finding attention, span representation from the concatenated contextualized representations of each word;
obtaining a mention score using a mention feed-forward neural network with a self-attention mechanism based on the generated span representation;
obtaining an antecedent score using an antecedent feed-forward neural network with the self-attention mechanism based on the generated span representation; and
obtaining a coreference score based on at least the generated mention score and the generated antecedent score; and
computing, using a masked softmax layer, probability distribution for each mention based at least on the coreference score.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4 wherein the probability distribution is only over candidate antecedents for each mention, with probability distribution for mentions after the current mention in the document masked by the masked softmax layer.

### Claim 6 (depends on 4)

The computer-implemented method of claim 4 wherein the self-attention mechanism averages over a previous iteration's representations weighted by a normalized coreference scores.

### Claim 7 (depends on 4)

The computer-implemented method of claim 4 wherein the generated span representations with probability scores less than a predetermined threshold are pruned from coreference decisions.

### Claim 8

A computer-implement method for coreference resolution using a coreference resolution model comprising:
receiving a document comprising a set of words;
generating, using a character and word embeddings encoder, a plurality of embeddings with each embedding as a concatenation of fixed word embeddings and convolutional neural network (CNN) character embeddings;
computing and concatenating, using a bidirectional Long short-term memory (LSTM) layer, contextualized representation of each word in the document from two directions;
performing iterative operations comprising:
generating, with head-finding attention, span representation from the concatenated contextualized representations for a current mention;
obtaining a mention score using a mention feed-forward neural network with a self-attention mechanism based on the generated span representation;
obtaining an antecedent score using an antecedent feed-forward neural network with the self-attention mechanism based on the generated span representation; and
obtaining a coreference score for the current mention based on at least the generated mention score and the generated antecedent score;
computing, using a masked softmax layer, probability distribution over a set of actions in which the set of actions comprise linking the current identified mention with a prior identified mention or not linking the current identified mention to any prior identified mention for the current mention based at least on the coreference score;
selecting an action from the set of actions using the probability distribution of actions; and
based upon the selected action, updating a coreference graph for the document, in which the coreference graph comprises mentions as nodes and links representing coreference connections between mentions.

### Claim 9 (depends on 8)

The computer-implement method of claim 8 wherein the a coreference resolution model is pretrained using steps comprising:
inputting a training document into the coreference resolution model to generate a set of coreference graphs and a corresponding set of trajectories of actions for each document in an iterative operation;
obtaining a sample set of coreference graphs from the set of coreference graphs;
computing a reward for each coreference graph from the sample set of coreference graphs;
using the rewards and trajectories of actions in the sample set to compute a gradient; and
using the gradient to update parameters of the coreference resolution model.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9 wherein the gradient further comprising an entropy regularization parameter to control exploration of the set of trajectories of actions.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10 wherein the set of trajectories of actions are sampled based on a current policy when the entropy regularization parameter is set as 0.

### Claim 12 (depends on 10)

The computer-implemented method of claim 10 wherein the set of trajectories of actions are sampled uniformly regardless of a current policy when the entropy regularization parameter is set as ∞.

### Claim 13 (depends on 9)

The computer-implemented method of claim 9 wherein the probability distribution is only over candidate antecedents for each mention, with probability distribution for mentions after the current mention in the document masked by the masked softmax layer.

### Claim 14 (depends on 9)

The computer-implemented method of claim 9 wherein the self-attention mechanism averages over a previous iteration's representations weighted by a normalized coreference scores.

### Claim 15

A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by one or more processors, causes the steps for training a coreference resolution model to be performed comprising:
[a] inputting a document comprising a set of text into a policy network to identify mentions in the document;
[b] given a current identified mention in the document, using the policy network to obtain a probability distribution of a set of actions in which the set of actions comprise linking the current identified mention with a prior identified mention or not linking the current identified mention to any prior identified mention;
[c] selecting an action from the set of actions using the probability distribution of actions;
[d] based upon the selected action, updating a coreference graph for the document, in which the coreference graph comprises mentions as nodes and links representing coreference connections between mentions;
[e] responsive to the document having another mention, selecting it as the current identified mention and returning to step [b];
[f] responsive to the document not having another mention, outputting the coreference graph for the document;
[g] using the outputted coreference graph and a ground truth coreference information for the document, computing a reward based upon one or more metrics;
[h] using a trajectory of actions selected and the reward to compute a gradient; and
[i] updating the policy network using the gradient.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium or media of claim 15 the policy network is pre-trained using training steps comprising:
inputting a set of documents into the policy network that identifies mentions in the documents and generates a coreference graph for each document;
using corresponding ground-truth coreference graphs for the document to compute a loss relative to the generated coreference graphs obtained from the policy network;
using the loss to update the policy network; and
iterating the above training steps until a stopping condition is reached, the step condition comprises one or more criteria from number of epochs, error level, or number of iterations.

### Claim 17 (depends on 16)

The non-transitory computer-readable medium or media of claim 16 further comprising:
repeating the steps of [a]-[f] for the document to obtain a set of coreference graphs and a corresponding set of trajectories of actions for each document in an iterative operation;
obtaining a sample set of coreference graphs from the set of coreference graphs;
computing a reward for each coreference graph from the sample set of coreference graphs; and
using the rewards and trajectories of actions in the sample set to compute a gradient.

### Claim 18 (depends on 15)

The non-transitory computer-readable medium or media of claim 15 wherein step [a] further comprising:
generating, using a character and word embeddings encoder, a plurality of embeddings with each embedding as a concatenation of fixed pretrained word embeddings and convolutional neural network (CNN) character embeddings;
computing and concatenating, using a bidirectional Long short-term memory (LSTM) layer, contextualized representation of each word in the input document from two directions;
performing iterative operations comprising:
generating, with head-finding attention, span representation from the concatenated contextualized representations of each word;
obtaining a mention score using a mention feed-forward neural network with a self-attention mechanism based on the generated span representation;
obtaining an antecedent score using an antecedent feed-forward neural network with the self-attention mechanism based on the generated span representation; and
obtaining a coreference score based on at least the generated mention score and the generated antecedent score; and
computing, using a masked softmax layer, probability distribution for each mention based at least on the coreference score.

### Claim 19 (depends on 18)

The non-transitory computer-readable medium or media of claim 18 wherein the probability distribution is only over candidate antecedents for each mention, with probability distribution for mentions after the current mention in the document masked by the masked softmax layer.

### Claim 20 (depends on 18)

The non-transitory computer-readable medium or media of claim 18 wherein the self-attention mechanism averages over a previous iteration's representations weighted by a normalized coreference scores.

### Claim 21 (depends on 1)

A system for training a coreference resolution model, comprising at least one processor, and a memory storing instructions, wherein the instructions when executed by the at least one processor, cause the at least one processor to perform the computer-implement method of claim 1.