Answer selection using a compare-aggregate model with language model and condensed similarity information from latent clustering

A computerized method comprising:

• determining, for a pair of a query and a candidate answer of a target corpus, probabilities of the query and the candidate answer being in each of a plurality of latent groupings of the target corpus;
• consolidating the probabilities by pruning out a set of the probabilities (i) corresponding to a set of the latent groupings having values of the probabilities below a designated threshold, or (ii) corresponding to a number of the latent groupings above a designated threshold;
• generating latent clustering information from the consolidated probabilities;
• and
• generating a measure of textual similarity between the query and the candidate answer using the latent clustering information as auxiliary information.

A computer system comprising at least one processor and memory comprising:

• a set of latent memory vectors representing latent clusters of a set of candidate answers of a target document;
• a textual-similarity computing model;
• and
• an answer selection component configured to select, from the set of candidate answers of the target document, an answer to a question using the textual-similarity computing model, the answer selection component comprising:
• a first latent cluster component configured to generate a first measure of similarity between the question and the set of latent memory vectors by dropping similarity information for latent clusters beyond a threshold of similarity to the question;
• and
• a second latent cluster component configured to generate a second measure of similarity between the candidate answer and the set of latent memory vectors by dropping similarity information for latent clusters beyond a threshold of similarity to the answer;
• wherein the textual-similarity computing model is configured to compute textual similarity based on the first and second measures of similarity.