Weighting and expanding query terms based on language model favoring surprising words
AssigneeInternational Business Machines Corporation
InventorsKyle L. Croutwater, Vikrant Verma, Le Zhang, Zhe Zhang
Life of this patent
Estimating the term…
Abstract
An approach is provided that receives a question at a question answering (QA) system. The question includes a number of words. The approach operates by calculating weights that correspond to search terms included in the plurality of words. The search terms include the plurality of words and may include terms that are one or more sequences of adjacent words included in the question. Based on the calculated weights and the words in the question, the approach generates a query that is used to search a corpus that is managed by the QA system with the search resulting in one or more search results.
Claims7
An information handling system comprising:
- one or more processors;
- a memory coupled to at least one of the processors;
- and
- a set of computer program instructions stored in the memory and executed by at least one of the processors in order to perform actions comprising:
- receiving a question at a question answering (QA) system that includes a plurality of words comprising a first word and a second word adjacent to the first word;
- retrieving a first log likelihood value corresponding to the first word, wherein the first log likelihood value is a derivative of a first probability that the first word resides in an n-gram corpus;
- retrieving a second log likelihood value corresponding to a phrase comprising the first word and the second word, wherein the second log likelihood value is a derivative of a second probability that the phrase comprising the first word and the second word resides in the n-gram corpus;
- calculating a weighting of the first word based on the first log likelihood value and the second log likelihood value;
- in response to determining that the weighting of the first word meets a first threshold:
- detecting that the first word is included in a list of removable common words to remove from one or more queries;
- and
- updating the list of removable common words by removing the first word from the list of removable common words;
- generating one of the one or more queries based on the question, the updated list of removable common words, and the calculated weighting;
- and
- searching, using the generated query, a corpus managed by the QA system, wherein the searching results in one or more search results.
A computer program product stored in a computer readable storage medium, comprising computer program code that, when executed by an information handling system, performs actions comprising:
- receiving a question at a question answering (QA) system that includes a plurality of words comprising a first word and a second word adjacent to the first word;
- retrieving a first log likelihood value corresponding to the first word, wherein the first log likelihood value is a derivative of a first probability that the first word resides in an n-gram corpus;
- retrieving a second log likelihood value corresponding to a phrase comprising the first word and the second word, wherein the second log likelihood value is a derivative of a second probability that the phrase comprising the first word and the second word resides in the n-gram corpus;
- calculating a weighting of the first word based on the first log likelihood value and the second log likelihood value;
- in response to determining that the weighting of the first word meets a first threshold:
- detecting that the first word is included in a list of removable common words to remove from one or more queries;
- and
- updating the list of removable common words by removing the first word from the list of removable common words;
- generating one of the one or more queries based on the question, the updated list of removable common words, and the calculated weighting;
- and
- searching, using the generated query, a corpus managed by the QA system, wherein the searching results in one or more search results.
Dependencies are inferred from the claim text.