# US 2022/0044093 A1 · Generating dual sequence inferences using a neural network model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044093 A1
- Document ID: US20220044093A1
- Status: Published application
- Kind: A1
- Application number: 17/506,033
- Filed: 2021-10-20
- Published: 2022-02-10
- Earliest claimed: 2017-10-27
- Assignees: salesforce.com, inc.
- Applicants: salesforce.com, inc.
- Inventors: Victor Zhong; Caiming Xiong; Richard Socher
- Web page: https://patentgrep.com/patent/US20220044093A1

## CPC

- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06F 16/2462: {Approximate or statistical queries}
- G06F 16/93: Document management systems
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 5/04: Inference or reasoning models
- G06N 3/048: Activation functions
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A computer-implemented method for dual sequence inference using a neural network model includes generating a codependent representation based on a first input representation of a first sequence and a second input representation of a second sequence using an encoder of the neural network model and generating an inference based on the codependent representation using a decoder of the neural network model. The neural network model includes a plurality of model parameters learned according to a machine learning process. The encoder includes a plurality of coattention layers arranged sequentially, each coattention layer being configured to receive a pair of layer input representations and generate one or more summary representations, and an output layer configured to receive the one or more summary representations from a last layer among the plurality of coattention layers and generate the codependent representation.

## Claims

### Claim 1

A computer-implemented method for training a neural network model, the method comprising:
generating, using the neural network model, a series of inferences from a first training sequence and a second training sequence;
generating a mixed learning objective from the series of inferences, wherein the generating further comprises:
determining, using a supervised learning objective, a first loss or reward for an inference in the series of inferences independently of other inferences in the series of inferences;
determining, using a reinforcement learning objective, a second loss or reward over the series of inferences; and
combining the supervised learning objective and the reinforcement learning objective into a mixed objective; and
updating parameters of the neural network model based on a loss or reward in the mixed learning objective.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the updating further comprises:
determining a gradient of the mixed learning objective with respect to the parameters; and
updating the parameters based on the gradient.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising:
determining the loss and reward of the mixed objective based on the series of inferences and a ground truth associated with the first training sequence or the second training sequence.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein a differentiable objective function determines the first loss or reward.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein the differentiable objective function is based on start and end positions of an inference answer span in the inference to a question in the first training sequence or the second training sequence and start and end positions of a ground truth answer span.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the determining the second loss or reward further comprises:
determining a baseline score using a scoring function;
determining a reinforcement learning reward function based on the scoring function and the baseline score; and
determining the second loss or reward based on the reinforcement learning reward function.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein the baseline score is based on baseline start and end positions of an answer span and ground truth start and end positions of the answer span; and
wherein the reinforcement learning reward function is based on the ground truth start and end positions of the answer span and a start and end positions of an inference answer span.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the first training sequence corresponds to a document, the second training sequence corresponds to a question, and the inference corresponds to a span of text in the document that answers the question.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the supervised learning objective is a binary evaluation objective and the reinforcement learning objective is a non-binary objective.

### Claim 10

A system for training a neural network model, the system comprising:
a memory configured to store the neural network model; and
a processor configured to:
generate, using the neural network model, a series of inferences from a first training sequence and a second training sequence;
generate a mixed learning objective from the series of inferences, wherein to generate the mixed learning objective the processor is further configured to:
determine, using a supervised learning objective, a first loss or reward for an inference in the series of inferences independently of other inferences in the series of inferences;
determine, using a reinforcement learning objective, a second loss or reward over the series of inferences; and
combine the supervised learning objective and the reinforcement learning objective into a mixed objective; and
update parameters of the neural network model based on the mixed learning objective.

### Claim 11 (depends on 10)

The system of claim 10, wherein to update the parameters, the processor is further configured to:
determine a gradient of the mixed learning objective with respect to the parameters; and
update the parameters based on the gradient.

### Claim 12 (depends on 10)

The system of claim 10, wherein the processor is further configured to:
determine a loss and reward of the mixed objective based on the series of inferences and a ground truth associated with the first training sequence or the second training sequence.

### Claim 13 (depends on 10)

The system of claim 10, wherein a differentiable objective function determines the first loss or reward.

### Claim 14 (depends on 13)

The system of claim 13, wherein the differentiable objective function is based on start and end positions of an inference answer span in the inference to a question in the first training sequence or the second training sequence and start and end positions of a ground truth answer span.

### Claim 15 (depends on 10)

The system of claim 10, wherein to determine the second loss or reward the processor is further configured to:
determine a baseline score using a scoring function;
determine a reinforcement learning reward function based on the scoring function and the baseline score; and
determine the second loss or reward based on the reinforcement learning reward function.

### Claim 16 (depends on 15)

The system of claim 15, wherein the baseline score is based on baseline start and end positions of an answer span and ground truth start and end positions of the answer span; and
wherein the reinforcement learning reward function is based on the ground truth start and end positions of the answer span and start and end positions of an inference answer span.

### Claim 17 (depends on 10)

The system of claim 10, wherein the first training sequence corresponds to a document, the second training sequence corresponds to a question, and the inference corresponds to a span of text in the document that answers the question.

### Claim 18 (depends on 10)

The system of claim 10, wherein the supervised learning objective is a binary evaluation objective and the reinforcement learning objective is a non-binary objective.

### Claim 19

A non-transitory computer readable medium having instructions stored thereon, that when executed by a processor cause the processor to perform operations for training a neural network model, the operations comprising:
generating, using the neural network model, a series of inferences from a training sequence;
generating a mixed learning objective from the series of inferences, wherein generating the mixed learning objective further comprises:
determining, using a supervised learning objective, a first loss or reward for an inference in the series of inferences independently of other inferences in the series of inferences;
determining, using a reinforcement learning objective, a second loss or reward over the series of inferences; and
combining the supervised learning objective and the reinforcement learning objective into a mixed objective; and
updating parameters of the neural network model based on the mixed learning objective.

### Claim 20 (depends on 19)

The non-transitory computer readable medium of claim 19, further comprising:
determining a gradient of the mixed learning objective with respect to the parameters; and
updating the parameters based on the gradient.
