# US 10,121,386 B2 · Learning model for dynamic component utilization in a question answering system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,121,386 B2 - Document ID: US10121386B2 - Status: Granted - Kind: B2 - Application number: 14/872,307 - Filed: 2015-10-01 - Published: 2016-01-21 - Granted: 2018-11-06 - Earliest claimed: 2013-03-15 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Corville O. Allen; Kyle L. Croutwater; Kevin B. Haverlock; Michael D. Whitley - Web page: https://patentgrep.com/patent/US10121386B2 ## CPC - G09B 7/00: Electrically-operated teaching apparatus or devices working with questions and answers (mechanically operated G09B3/00; computing arrangements G06F) - G09B 7/02: of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student - G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B) ## Abstract Mechanisms are provided in a data processing system for utilizing algorithms based on categories in a question answering system. The mechanisms capture a history of performance and correctness metrics for identifying efficiency of respective algorithms for finding answers to questions in respective question categories in a question answering system. The mechanisms determine sets of algorithms to use for respective question categories according to efficiency and correctness analysis. The mechanisms determine a question category of a given input question and execute a set of algorithms corresponding to the question category of the given input question that meet an efficiency threshold to contribute to finding a correct answer for the given input question. ## Claims ### Claim 1 A method, in a data processing system having a processor and a memory, wherein the memory comprises instructions which are executed by the processor to cause the processor to implement an algorithm utilization controller for utilizing algorithm components based on categories in a question answering system, the method comprising: capturing, by a question answering engine executing within the question answering system, a history of performance and correctness metrics for identifying efficiency of respective algorithms for finding answers to a plurality of training questions in respective question categories in the question answering system, wherein the algorithms comprise specialized software components that perform portions of analysis performed by the question answering engine; determining, by the algorithm utilization controller, sets of algorithms to use for respective question categories according to efficiency and correctness analysis, wherein the algorithm utilization controller comprises a logical grouping component, a learning analyzer component, a machine learning component, and an algorithm execution broker component working together to dynamically adjust algorithms to answer questions while minimizing resources, wherein determining the sets of algorithms comprises classifying each given training question into a question category by the logical grouping component, profiling resources to produce a performance profile for the question category for output value contributions to produce a final answer by the learning analyzer component, utilizing the performance profile for the question category as training data by the machine learning component considering confidence of answers as a weighting, and generating a machine learning model for classifying questions and adjusting algorithms dynamically by the algorithm execution broker according to a criteria adjusted for resource availability and answer correctness criteria; for a given input question, determining, by the algorithm utilization controller, a question category of the given input question using the machine learning model; determining, by the algorithm utilization controller, a set of algorithms corresponding to the question category of the given input question that meet an efficiency threshold to contribute to finding a correct answer for the given input question using the machine learning model; and executing, by the question answering engine, the set of algorithms corresponding to the question category of the given input question to find an answer to the given input question. ### Claim 2 (depends on 1) The method of claim 1, wherein capturing the history of performance and correctness metrics further comprises: recording performance characteristics for each of a plurality of algorithms; and generating a performance profile for each of the plurality of algorithms for each question category. ### Claim 3 (depends on 2) The method of claim 2, wherein the performance characteristics comprise heap sizes, CPU utilization, memory usage, the execution time of an algorithm, file input and output access, or write speeds. ### Claim 4 (depends on 1) The method of claim 1, wherein determining sets of algorithms to use for respective question categories comprises: making correlations between algorithms and logical groupings to provide a category-specific machine learning model. ### Claim 5 (depends on 4) The method of claim 4, further comprising determining which algorithms to execute and necessary system resources using a category-specific machine learning model of the determined question category. ### Claim 6 (depends on 1) The method of claim 1, wherein the correctness metrics comprise a number of features for each algorithm produced for a given logical grouping and how features for that logical grouping impact each question category. ### Claim 7 A computer program product comprising a non-transitory computer readable medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to implement an algorithm utilization controller for utilizing algorithm components based on categories in a question answering system, wherein the computer readable program causes the computing device to: capture, by a question answering engine executing within the question answering system, a history of performance and correctness metrics for identifying efficiency of respective algorithms for finding answers to a plurality of training questions in respective question categories in the question answering system, wherein the algorithms comprise specialized software components that perform portions of analysis performed by the question answering engine; determine, by the algorithm utilization controller, sets of algorithms to use for respective question categories according to efficiency and correctness analysis, wherein the algorithm utilization controller comprises a logical grouping component, a learning analyzer component, a machine learning component, and an algorithm execution broker component working together to dynamically adjust algorithms to answer questions while minimizing resources, wherein determining the sets of algorithms comprises classifying each given training question into a question category by the logical grouping component, profiling resources to produce a performance profile for the question category for output value contributions to produce a final answer by the learning analyzer component, utilizing the performance profile for the question category as training data by the machine learning component considering confidence of answers as a weighting, and generating a machine learning model for classifying questions and adjusting algorithms dynamically by the algorithm execution broker according to a criteria adjusted for resource availability and answer correctness criteria; for a given input question, determine, by the algorithm utilization controller, a question category of the given input question using the machine learning model; determining, by the algorithm utilization controller, a set of algorithms corresponding to the question category of the given input question that meet an efficiency threshold to contribute to finding a correct answer for the given input question using the machine learning model; and execute, by the question answering engine, the set of algorithms corresponding to the question category of the given input question to find an answer to the given input question. ### Claim 8 (depends on 7) The computer program product of claim 7, wherein capturing the history of performance and correctness metrics further comprises: recording performance characteristics for each of a plurality of algorithms; and generating a performance profile for each of the plurality of algorithms for each question category. ### Claim 9 (depends on 8) The computer program product of claim 8, wherein the performance characteristics comprise heap sizes, CPU utilization, memory usage, the execution time of an algorithm, file input and output access, or write speeds. ### Claim 10 (depends on 7) The computer program product of claim 7, wherein determining sets of algorithms to use for respective question categories comprises: making correlations between algorithms and logical groupings to provide a category-specific machine learning model. ### Claim 11 (depends on 10) The computer program product of claim 10, wherein executing the set of algorithms corresponding to the question category of the given input question comprises determining which algorithms to execute and necessary system resources using a category-specific machine learning model of the determined question category. ### Claim 12 (depends on 7) The computer program product of claim 7, wherein the computer readable program is stored in a computer readable storage medium in a data processing system and wherein the computer readable program was downloaded over a network from a remote data processing system. ### Claim 13 (depends on 7) The computer program product of claim 7, wherein the computer readable program is stored in a computer readable storage medium in a server data processing system and wherein the computer readable program is downloaded over a network to a remote data processing system for use in a computer readable storage medium with the remote system. ### Claim 14 (depends on 7) The computer program product of claim 7, wherein the correctness metrics comprise a number of features for each algorithm produced for a given logical grouping and how features for that logical grouping impact each question category. ### Claim 15 An apparatus, comprising: a processor; and a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to implement an algorithm utilization controller for utilizing algorithm components based on categories in a question answering system, wherein the instructions cause the processor to: capture, by a question answering engine executing within the question answering system, a history of performance and correctness metrics for identifying efficiency of respective algorithms for finding answers to a plurality of training questions in respective question categories in the question answering system, wherein the algorithms comprise specialized software components that perform portions of analysis performed by the question answering engine; determine, by the algorithm utilization controller, sets of algorithms to use for respective question categories according to efficiency and correctness analysis, wherein the algorithm utilization controller comprises a logical grouping component, a learning analyzer component, a machine learning component, and an algorithm execution broker component working together to dynamically adjust algorithms to answer questions while minimizing resources, wherein determining the sets of algorithms comprises classifying each given training question into a question category by the logical grouping component, profiling resources to produce a performance profile for the question category for output value contributions to produce a final answer by the learning analyzer component, utilizing the performance profile for the question category as training data by the machine learning component considering confidence of answers as a weighting, and generating a machine learning model for classifying questions and adjusting algorithms dynamically by the algorithm execution broker according to a criteria adjusted for resource availability and answer correctness criteria; for a given input question, determine, by the algorithm utilization controller, a question category of the given input question using the machine learning model; determining, by the algorithm utilization controller, a set of algorithms corresponding to the question category of the given input question that meet an efficiency threshold to contribute to finding a correct answer for the given input question using the machine learning model; and execute, by the question answering engine, the set of algorithms corresponding to the question category of the given input question to find an answer to the given input question. ### Claim 16 (depends on 15) The apparatus of claim 15, wherein capturing the history of performance and correctness metrics further comprises: recording performance characteristics for each of a plurality of algorithms; and generating a performance profile for each of the plurality of algorithms for each question category. ### Claim 17 (depends on 16) The apparatus of claim 16, wherein the performance characteristics comprise heap sizes, CPU utilization, memory usage, the execution time of an algorithm, file input and output access, or write speeds. ### Claim 18 (depends on 15) The apparatus of claim 15, wherein determining sets of algorithms to use for respective question categories comprises: making correlations between algorithms and logical groupings to provide a category-specific machine learning model. ### Claim 19 (depends on 18) The apparatus of claim 18, wherein executing the set of algorithms corresponding to the question category of the given input question comprises determining which algorithms to execute and necessary system resources using a category-specific machine learning model of the determined question category. ### Claim 20 (depends on 15) The apparatus of claim 15, wherein the correctness metrics comprise a number of features for each algorithm produced for a given logical grouping and how features for that logical grouping impact each question category.