Abstractive multi-document summarization through self-supervision and control

A method of generating enriched training data for a multi-source transformer neural network for generation of a summary of one or more passages of input text using a processor, the method comprising:

• creating, from a plurality of input text sets, training points each comprising an input text subset of the input text set and a corresponding reference input text from the input text set, wherein creating the training points comprises providing a similarity between the reference input text and the input text subset, wherein the size of the input text subset is a predetermined number;
• for each created training point, selecting one or more reference features corresponding to categorical labels of the respective reference input text;
• selecting, for each input text in the input text subsets of the training points, control codes from the selected reference features;
• and
• enriching the input texts in the input text subsets of the training points with the selected control codes to generate enriched training data.

A machine-learning system implemented using a processor for generating enriched training data for a multi-source transformer neural network for summarizing input text, the system comprising:

• a relevance analysis module configured to create, from an input text set, training points comprising a reference input text from the input text set and an input text subset of the input text set, wherein creating the training point comprises selecting an input text as reference input text and optimizing a similarity of the corresponding input text subset;
• a plurality of classifiers for classifying whether an input text falls under a categorical label;
• and
• processor-based means for collecting, for each training point, weights for reference features from the plurality of trained classifiers, comprising selecting the reference features from the trained classifiers corresponding to categorical labels of the respective reference input text and for selecting, for each input text in the input text subsets of the training points, control codes from the reference features and enriching the input text with the respectively selected control codes to generate enriched training data.

A non-transitory computer-readable medium having executable instructions stored thereon, the executable instructions when read causing a processor to perform a method for generating enriched training data for a multi-source transformer neural network for generation of a summary of one or more passages of input text, the method comprising:

• creating, from a plurality of input text sets, training points each comprising an input text subset of the input text set and a corresponding reference input text from the input text set, wherein creating the training points comprises providing a similarity between the reference input text and the input text subset, wherein the size of the input text subset is a predetermined number;
• training a plurality of classifiers for a plurality of categorical labels, each classifier classifying whether an input text falls under a categorical label from the plurality of categorical labels;
• collecting, for each training point, weights for reference features from the plurality of trained classifiers, comprising selecting the reference features from the trained classifiers corresponding to categorical labels of the respective reference input text;
• selecting, for each input text in the input text subsets of the training points, based on the weights for the reference features, control codes from the reference features;
• and
• enriching the input text with the selected control codes to generate enriched training data.