# US 12,217,146 B2 · Generating dual sequence inferences using a neural network model > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,146 B2 - Document ID: US12217146B2 - Status: Granted - Kind: B2 - Application number: 17/506,033 - Filed: 2021-10-20 - Published: 2022-02-10 - Granted: 2025-02-04 - Earliest claimed: 2017-10-27 - Assignees: Salesforce, Inc. - Applicants: Salesforce, Inc. - Inventors: Victor Zhong; Caiming Xiong; Richard Socher - Web page: https://patentgrep.com/patent/US12217146B2 ## CPC - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06F 16/2462: {Approximate or statistical queries} - G06F 16/93: Document management systems - G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 3/092: Reinforcement learning - G06N 5/04: Inference or reasoning models - G06N 3/048: Activation functions - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A computer-implemented method for dual sequence inference using a neural network model includes generating a codependent representation based on a first input representation of a first sequence and a second input representation of a second sequence using an encoder of the neural network model and generating an inference based on the codependent representation using a decoder of the neural network model. The neural network model includes a plurality of model parameters learned according to a machine learning process. The encoder includes a plurality of coattention layers arranged sequentially, each coattention layer being configured to receive a pair of layer input representations and generate one or more summary representations, and an output layer configured to receive the one or more summary representations from a last layer among the plurality of coattention layers and generate the codependent representation. ## Claims ### Claim 1 A computer-implemented method for training a neural network model, the method comprising: generating, using the neural network model, a series of inferences from a first training sequence and a second training sequence; generating a mixed learning objective from the series of inferences, wherein the generating further comprises: determining, using a supervised learning objective, a first loss or a first reward for an inference in the series of inferences independently of other inferences in the series of inferences; determining, using a reinforcement learning objective, a second loss or a second reward over the series of inferences by: determining a baseline score using a scoring function, wherein the baseline score is based on baseline start and end positions of an answer span and a ground truth start and end positions of the answer span; determining a reinforcement learning reward function based on the scoring function and the baseline score, wherein the reinforcement learning reward function is based on the ground truth start and end positions of the answer span and a start and an end positions of an inference answer span; and determining the second loss or the second reward based on the reinforcement learning reward function; and combining the supervised learning objective and the reinforcement learning objective into a mixed objective using the first and second loss or the first and second reward; and updating parameters of the neural network model based on a loss or a reward in the mixed learning objective. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the updating further comprises: determining a gradient of the mixed learning objective with respect to the parameters; and updating the parameters based on the gradient. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising: determining the loss or the reward of the mixed objective based on the series of inferences and a ground truth associated with the first training sequence or the second training sequence. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein a differentiable objective function determines the first loss or the first reward. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein the differentiable objective function is based on the start and the end positions of the inference answer span in an inference to a question in the first training sequence or the second training sequence and the start and the end positions of the ground truth answer span. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the first training sequence corresponds to a document, the second training sequence corresponds to a question, and the inference corresponds to a span of text in the document that answers the question. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the supervised learning objective is a binary evaluation objective and the reinforcement learning objective is a non-binary objective. ### Claim 8 A system for training a neural network model, the system comprising: a memory configured to store the neural network model; and a processor configured to: generate, using the neural network model, a series of inferences from a first training sequence and a second training sequence; generate a mixed learning objective from the series of inferences, wherein to generate the mixed learning objective the processor is further configured to: determine, using a supervised learning objective, a first loss or a first reward for an inference in the series of inferences independently of other inferences in the series of inferences; determine, using a reinforcement learning objective, a second loss or a second reward over the series of inferences by: determine a baseline score using a scoring function, wherein the baseline score is based on baseline start and end positions of an answer span and a ground truth start and end positions of the answer span; determine a reinforcement learning reward function based on the scoring function and the baseline score, wherein the reinforcement learning reward function is based on the ground truth start and end positions of the answer span and a start and an end positions of an inference answer span; and determine the second loss or the second reward based on the reinforcement learning reward function; and combine the supervised learning objective and the reinforcement learning objective into a mixed objective using the first and second loss or the first and second reward; and update parameters of the neural network model based on the mixed learning objective. ### Claim 9 (depends on 8) The system of claim 8, wherein to update the parameters, the processor is further configured to: determine a gradient of the mixed learning objective with respect to the parameters; and update the parameters based on the gradient. ### Claim 10 (depends on 8) The system of claim 8, wherein the processor is further configured to: determine a loss and reward of the mixed objective based on the series of inferences and a ground truth associated with the first training sequence or the second training sequence. ### Claim 11 (depends on 8) The system of claim 8, wherein a differentiable objective function determines the first loss or the first reward. ### Claim 12 (depends on 11) The system of claim 11, wherein the differentiable objective function is based on the start and the end positions of the inference answer span in an inference to a question in the first training sequence or the second training sequence and start and end positions of the ground truth answer span. ### Claim 13 (depends on 8) The system of claim 8, wherein the first training sequence corresponds to a document, the second training sequence corresponds to a question, and the inference corresponds to a span of text in the document that answers the question. ### Claim 14 (depends on 8) The system of claim 8, wherein the supervised learning objective is a binary evaluation objective and the reinforcement learning objective is a non-binary objective. ### Claim 15 A non-transitory computer readable medium having instructions stored thereon, that when executed by a processor cause the processor to perform operations for training a neural network model, the operations comprising: generating, using the neural network model, a series of inferences from a training sequence; generating a mixed learning objective from the series of inferences, wherein generating the mixed learning objective further comprises: determining, using a supervised learning objective, a first loss or a first reward for an inference in the series of inferences independently of other inferences in the series of inferences; determining, using a reinforcement learning objective, a second loss or a second reward over the series of inferences by: determining a baseline score using a scoring function, wherein the baseline score is based on baseline start and end positions of an answer span and a ground truth start and end positions of the answer span; determining a reinforcement learning reward function based on the scoring function and the baseline score, wherein the reinforcement learning reward function is based on the ground truth start and end positions of the answer span and a start and end positions of an inference answer span; and determining the second loss or the second reward based on the reinforcement learning reward function; and combining the supervised learning objective and the reinforcement learning objective into a mixed objective using the first and second loss or the first and second reward; and updating parameters of the neural network model based on the mixed learning objective. ### Claim 16 (depends on 15) The non-transitory computer readable medium of claim 15, further comprising: determining a gradient of the mixed learning objective with respect to the parameters; and updating the parameters based on the gradient. ### Claim 17 (depends on 15) The non-transitory computer readable medium of claim 15, wherein the updating further comprises: determining a gradient of the mixed learning objective with respect to the parameters; and updating the parameters based on the gradient. ### Claim 18 (depends on 15) The non-transitory computer readable medium of claim 15, further comprising: determining the loss or the reward of the mixed objective based on the series of inferences and a ground truth associated with the training sequence. ### Claim 19 (depends on 15) The non-transitory computer readable medium of claim 15, wherein the training sequence corresponds to a document or to a question, and the inference corresponds to a span of text in the document that answers the question. ### Claim 20 (depends on 15) The non-transitory computer readable medium of claim 15, wherein the supervised learning objective is a binary evaluation objective and the reinforcement learning objective is a non-binary objective.