# US 10,372,819 B2 · Determining off-topic questions in a question answering system using probabilistic language models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,372,819 B2 - Document ID: US10372819B2 - Status: Granted - Kind: B2 - Application number: 14/665,967 - Filed: 2015-03-23 - Published: 2016-09-29 - Granted: 2019-08-06 - Earliest claimed: 2015-03-23 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: John P. Bufe; Srinivasa Phani K. Gadde; Julius Goth, III - Web page: https://patentgrep.com/patent/US10372819B2 ## CPC - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G06F 40/289: Phrasal analysis, e.g. finite state techniques or chunking - G06F 40/295: Named entity recognition - G06F 40/30: Semantic analysis - G06N 5/045: Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A question answering system that determines whether a question is off-topic by performing the following steps: (i) receiving, by a question answering system, a set of documents; (ii) identifying topical subset(s) for each document of the set of documents using named entity recognition, where each topical subset relates to a corresponding topic; (iii) assigning a set of topic score(s) for each topical subset using natural language processing, where each topic score relates to a corresponding probability associated with the respective topical subset under a probabilistic language model; and (iv) determining, based, at least in part, on the topic score(s) corresponding to the topical subset(s), whether a question input into the question answering system is off-topic. ## Claims ### Claim 1 A computer program product comprising a computer readable storage medium having stored thereon: program instructions programmed to generate, for a question answering system adapted to answer questions for a particular domain, an on-topic probabilistic language model and an off-topic probabilistic language model, wherein the on-topic probabilistic language model is generated using training questions that have been identified as being on-topic with respect to the particular domain, and wherein the off-topic probabilistic language model is generated using training questions that have been identified as being off-topic with respect to the particular domain; program instructions programmed to determine, for a question input into the question answering system: (i) a first logarithm probability associated with the question under the on-topic probabilistic language model, and (ii) a second logarithm probability associated with the question under the off-topic probabilistic language model; program instructions programmed to model, using a sigmoid function, a difference between the first logarithm probability and the second logarithm probability; program instructions programmed to assign, based, at least in part, on the modelled difference, a topic score to the question; and program instructions programmed to determine, based, at least in part, on the topic score assigned to the question, whether the questions is off-topic with respect to the particular domain. ### Claim 2 (depends on 1) The computer program product of claim 1, wherein determining whether the question is off-topic with respect to the particular domain includes comparing the question's assigned topic score to a relevance score associated with an answer generated by the question answering system in response to the question. ### Claim 3 (depends on 2) The computer program product of claim 2, wherein: the question is determined to be off-topic if the question's assigned topic score is greater than the generated answer's associated relevance score; and the question is determined to be on-topic if the question's assigned topic score is less than the generated answer's associated relevance score. ### Claim 4 (depends on 1) The computer program product of claim 1, the computer readable storage medium having further stored thereon: program instructions programmed to determine to not answer the question based on the question being off-topic. ### Claim 5 (depends on 4) The computer program product of claim 4, the computer readable storage medium having further stored thereon: program instructions programmed to provide a response to a user who submitted the question informing the user that the question is off-topic, wherein the response prompts the user to ask a question that is not off-topic. ### Claim 6 (depends on 5) The computer program product of claim 5, the computer readable storage medium having further stored thereon: program instructions programmed to, in response to determining that an additional question input into the question answering system by the user is not off-topic, answer the additional question. ### Claim 7 (depends on 1) The computer program product of claim 1, the computer readable storage medium having further stored thereon: program instructions programmed to, in response to determining that the question is not off-topic, answer the question. ### Claim 8 (depends on 7) The computer program product of claim 7, wherein the modeling using the sigmoid function and the answering of the question are performed by an answerer module of a question answering pipeline of the question answering system. ### Claim 9 (depends on 8) The computer program product of claim 8, the computer readable storage medium having further stored thereon: program instructions programmed to integrate search and linguistic features derived from deep semantic knowledge of the question text into the answerer module. ### Claim 10 A computer system comprising: a processor(s) set; and a computer readable storage medium; wherein: the processor set is structured, located, connected and/or programmed to run program instructions stored on the computer readable storage medium; and the stored program instructions include: program instructions programmed to generate, for a question answering system adapted to answer questions for a particular domain, an on-topic probabilistic language model and an off-topic probabilistic language model, wherein the on-topic probabilistic language model is generated using training questions that have been identified as being on-topic with respect to the particular domain, and wherein the off-topic probabilistic language model is generated using training questions that have been identified as being off-topic with respect to the particular domain; program instructions programmed to determine, for a question input into the question answering system: (i) a first logarithm probability associated with the question under the on-topic probabilistic language model, and (ii) a second logarithm probability associated with the question under the off-topic probabilistic language model; program instructions programmed to model, using a sigmoid function, a difference between the first logarithm probability and the second logarithm probability; program instructions programmed to assign, based, at least in part, on the modelled difference, a topic score to the question; and program instructions programmed to determine, based, at least in part, on the topic score assigned to the question, whether the questions is off-topic with respect to the particular domain. ### Claim 11 (depends on 10) The computer system of claim 10, wherein: determining whether the question is off-topic with respect to the particular domain includes comparing the question's assigned topic score to a relevance score associated with an answer generated by the question answering system in response to the question; the question is determined to be off-topic if the question's assigned topic score is greater than the generated answer's associated relevance score; and the question is determined to be on-topic if the question's assigned topic score is less than the generated answer's associated relevance score. ### Claim 12 (depends on 10) The computer system of claim 10, wherein the stored program instructions further include: program instructions programmed to determine to not answer the question based on the question being off-topic. ### Claim 13 (depends on 12) The computer system of claim 12, wherein the stored program instructions further include: program instructions programmed to provide a response to a user who submitted the question informing the user that the question is off-topic, wherein the response prompts the user to ask a question that is not off-topic. ### Claim 14 (depends on 13) The computer system of claim 13, wherein the stored program instructions further include: program instructions programmed to, in response to determining that an additional question input into the question answering system by the user is not off-topic, answer the additional question. ### Claim 15 (depends on 10) The computer system of claim 10, wherein the stored program instructions further include: program instructions programmed to, in response to determining that the question is not off-topic, answer the question. ### Claim 16 (depends on 15) The computer system of claim 15, wherein the modeling using the sigmoid function and the answering of the question are performed by an answerer module of a question answering pipeline of the question answering system. ### Claim 17 (depends on 15) The computer system of claim 15, wherein the stored program instructions further include: program instructions programmed to integrate search and linguistic features derived from deep semantic knowledge of the question text into the answerer module.