Retrieval augmented clarification in interactive systems

A computer program product, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform:

• training a machine learning model to generate a clarification prompt in an interactive system, wherein the training further comprises:
• modifying an original query q in an existing question-answering dataset to generate a modified query q′, wherein the modified query q′ further comprises an under-specified query that is unclear based on modification of the original query;
• retrieving a top-K list of passages associated with the original query q and the modified query q′;
• determining, based on the retrieved top-K list of passages, that the modified query q meets a first criteria and a second criteria, wherein the first criteria is met in response to a determination that the top-K list of passages retrieved using the modified query q′is significantly different than the top-K list of passages retrieved using the original query q based on a certain threshold, and wherein the second criteria is met based on another determination that the modified query q′is a natural language question and at least one passage from the top-K list of passages associated with the modified query q′ comprises a correct answer to the original query q;
• based on the determination that the modified query q′ meets the first criteria and the second criteria, extracting one or more properties from the top-K list of passages associated with the modified query q′;
• selecting, using a computed disambiguation ability as a metric, at least one property from the extracted one or more properties to use as a differentiating factor that best distinguishes retrieved content candidates from the top-K list of passages, wherein the computed disambiguation ability quantifies a success of a given one of the extracted properties in distinguishing between the retrieved content candidates and is quantified using information gain;
• and
• training the machine learning model using a training dataset comprising the original query q, the modified query q′, and the differentiating factor, to generate the clarification prompt via an interactive user interface associated with the interactive system to clarify an intent of the user based on an actual under-specified query received from the user.

A system comprising a processor, connected to a memory, operable to perform:

• training a machine learning model to generate a clarification prompt in an interactive system, wherein the training further comprises:
• modifying an original query q in an existing question-answering dataset to generate a modified query q′, wherein the modified query q′ further comprises an under-specified query that is unclear based on modification of the original query;
• retrieving a top-K list of passages associated with the original query q and the modified query q′;
• determining, based on the retrieved top-K list of passages, that the modified query q meets a first criteria and a second criteria, wherein the first criteria is met in response to a determination that the top-K list of passages retrieved using the modified query q′is significantly different than the top-K list of passages retrieved using the original query q based on a certain threshold, and wherein the second criteria is met based on another determination that the modified query q′is a natural language question and at least one passage from the top-K list of passages associated with the modified query q′ comprises a correct answer to the original query q;
• based on the determination that the modified query q′ meets the first criteria and the second criteria, extracting one or more properties from the top-K list of passages associated with the modified query q′;
• selecting, using a computed disambiguation ability as a metric, at least one property from the extracted one or more properties to use as a differentiating factor that best distinguishes retrieved content candidates from the top-K list of passages, wherein the computed disambiguation ability quantifies a success of a given one of the extracted properties in distinguishing between the retrieved content candidates and is quantified using information gain;
• and
• training the machine learning model using a training dataset comprising the original query q, the modified query q′, and the differentiating factor, to generate the clarification prompt via an interactive user interface associated with the interactive system to clarify an intent of the user based on an actual under-specified query received from the user.