Search indexing using discourse trees

A system comprising:

• a non-transitory computer-readable medium storing computer-executable program instructions;
• and
• a processing device communicatively coupled to the non-transitory computer-readable medium for executing the computer-executable program instructions, wherein executing the computer-executable program instructions configures the processing device to perform operations comprising:
• creating, for an answer of a question-answer index comprising fragments of text, a discourse tree, wherein the discourse tree comprises a plurality of nodes, each non-terminal node representing a rhetorical relationship between two of the fragments of text and each terminal node is associated with one or more fragments of text;
• for each non-terminal node in the discourse tree:
• identifying, from the discourse tree, a rhetorical relationship associated with the non-terminal node,
• accessing a rule corresponding to the identified rhetorical relationship, wherein the rule identifies for selection, based on the identified rhetorical relationship, either a corresponding nucleus elementary discourse unit or a corresponding satellite elementary discourse unit, and
• selecting, based on the rule, one or more of (i) the fragment of text associated with the nucleus elementary discourse unit or (ii) the fragment of text associated with the satellite elementary discourse unit;
• and
• creating an index of additional questions comprising an entry corresponding to the selected fragment of text.

A non-transitory computer-readable storage medium storing computer-executable program instructions, wherein when executed by a processing device, the computer-executable program instructions cause the processing device to perform operations comprising:

• training a classification model by iteratively:
• providing one of a set of training data pairs to the classification model, wherein each training data pair comprises text and an expected classification of one or more of:
• a nucleus elementary discourse unit of a training discourse tree or a satellite elementary discourse unit of the training discourse tree;
• receiving, from the classification model, a determined classification that is indicative of either a nucleus elementary discourse unit or a satellite elementary discourse unit;
• calculating a loss function by calculating a difference between the determined classification and the expected classification;
• and
• adjusting internal parameters of the classification model to minimize the loss function;
• creating a discourse tree from fragments of an additional text, wherein the discourse tree comprises a plurality of nodes, each non-terminal node representing a rhetorical relationship between two of the fragments of the additional text and each terminal node is associated with one or more of the fragments of the additional text;
• and
• for each terminal node in the discourse tree, labeling the terminal node as either a nucleus elementary discourse unit or a satellite elementary discourse unit by applying the trained classification model to fragments of the additional text associated with one or more of the terminal nodes.