Method and system for long-form answer extraction based on combination of sentence index generation techniques

A system, comprising:

• a memory storing instructions;
• one or more communication interfaces;
• and
• one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to:
• receive a plurality of inputs, wherein the plurality of inputs is associated with information regarding a dataset comprising a question, a document comprising an answer text for the question;
• pre-process the plurality of inputs to obtain a set of pre-processed training data, wherein the set of pre-processed training data comprises a plurality of pre-processed sentence indices data and a plurality of pre-processed sentence index spans, wherein the plurality of pre-processed sentence indices data obtained using a document truncation technique, wherein the document truncation technique includes an entire answer text span from the document and extending the text span by adding sentences appearing immediately before and after the span as long as the extended span does not exceed a pre-defined number of tokens, wherein a sentence index of the plurality of sentence indices is appended at the beginning of each sentence in the document for the appending the sentence index;
• train a set of generation models using the set of pre-processed training data based on a supervised learning technique, where the set of models comprises a sentence indices generation model and a sentence index spans generation model, wherein the set of models are sequence-to-sequence models based on neural networks, and wherein:
• the sentence indices generation model comprises a plurality of answer sentence indices and the sentence index spans generation model comprises a plurality of answer sentence index spans, wherein the sentence indices generation model trained using the plurality of pre-processed sentence indices data, and the sentence index spans generation model trained using the plurality of pre-processed sentence index spans, wherein the set of generation models are generative auto-regressive sequence-to-sequence models use chain rule and model the probability of each token oi in the output sequence o conditioned on the input sequence x and the previously generated tokens o<I, wherein the set of generation models are trained by maximizing the loglikelihood of the input-output sequences in the training data;
• and
• post-process the plurality of answer sentence indices and the plurality of answer sentence index spans of the set of generation models for a long-form answer extraction, wherein the long form answer text is extracted for a user question input, based on mapping of a union of the first sentence indices and the second sentence indices, wherein the post-processing comprises:
• obtaining a first sentence indices by post-processing the plurality of answer sentence indices based on a cleaning technique;
• and
• obtaining a second sentence indices by post-processing the plurality of answer sentence index spans based on the cleaning technique and a span expansion technique.

One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:

• receiving a plurality of inputs, wherein the plurality of inputs is associated with information regarding a dataset comprising a question, a document comprising an answer text for the question;
• pre-processing the plurality of inputs to obtain a set of pre-processed training data, wherein the set of pre-processed training data comprises a plurality of pre-processed sentence indices data and a plurality of pre-processed sentence index spans, wherein the plurality of pre-processed sentence indices data obtained using a document truncation technique, wherein the document truncation technique includes an entire answer text span from the document and extending the text span by adding sentences appearing immediately before and after the span as long as the extended span does not exceed a pre-defined number of tokens, wherein a sentence index of the plurality of sentence indices is appended at the beginning of each sentence in the document for the appending the sentence index;
• training a set of generation models using the set of pre-processed training data based on a supervised learning technique, where the set of models comprises a sentence indices generation model and a sentence index spans generation model, wherein the set of models are sequence-to-sequence models based on neural networks, and wherein the sentence indices generation model comprises a plurality of answer sentence indices and the sentence index spans generation model comprises a plurality of answer sentence index spans, wherein the sentence indices generation model trained using the plurality of pre-processed sentence indices data, and the sentence index spans generation model trained using the plurality of pre-processed sentence index spans, wherein the set of generation models are generative auto-regressive sequence-to-sequence models use chain rule and model the probability of each token oi in the output sequence o conditioned on the input sequence x and the previously generated tokens o<I, wherein the set of generation models are trained by maximizing the loglikelihood of the input-output sequences in the training data;
• and
• post-processing the plurality of answer sentence indices and the plurality of answer sentence index spans of the set of generation models for a long-form answer extraction, via the one or more hardware processors, wherein the long form answer text is extracted for a user question input, based on mapping of a union of the first sentence indices and the second sentence indices, wherein the post-processing comprises:
• obtaining a first sentence indices by post-processing the plurality of answer sentence indices based on a cleaning technique;
• and
• obtaining a second sentence indices by post-processing the plurality of answer sentence index spans based on the cleaning technique, and a span expansion technique.