# US 2019/0377984 A1 · Detecting suitability of machine learning models for datasets > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0377984 A1 - Document ID: US20190377984A1 - Status: Published application - Kind: A1 - Application number: 16/001,904 - Filed: 2018-06-06 - Published: 2019-12-12 - Earliest claimed: 2018-06-06 - Assignees: DataRobot, Inc. - Applicants: DataRobot, Inc. - Inventors: SINDHU GHANTA; DREW ROSELLI; NISHA TALAGALA; VINAY SRIDHAR; SWAMINATHAN SUNDARARAMAN; LIOR AMAR; LIOR KHERMOSH; BHARATH RAMSUNDAR; SRIRAM SUBRAMANIAN - Web page: https://patentgrep.com/patent/US20190377984A1 ## CPC - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)} - G06F 18/2155: {characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling} - G06F 18/2193: {based on specific statistical tests} - G06N 5/04: Inference or reasoning models - G06N 20/00: Machine learning - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Apparatuses, systems, program products, and method are disclosed for detecting suitability of machine learning models for datasets. An apparatus includes a training evaluation module configured to calculate a first statistical data signature for a training data set of a machine learning system using one or more predefined statistical algorithms. An apparatus includes an inference evaluation module configured to calculate a second statistical data signature for an inference data set of a machine learning system using one or more predefined statistical algorithms. An apparatus includes a score module configured to calculate a suitability score describing the suitability of a training data set to an inference data set as a function of a first and a second statistical data signature. An apparatus includes an action module configured to perform an action related to a machine learning system in response to a suitability score satisfying an unsuitability threshold. ## Claims ### Claim 1 An apparatus comprising: a training evaluation module configured to calculate a first statistical data signature for a training data set of a machine learning system using one or more predefined statistical algorithms, the training data set used to generate a machine learning model; an inference evaluation module configured to calculate a second statistical data signature for an inference data set of the machine learning system using the one or more predefined statistical algorithms, the inference data set analyzed using the machine learning model; a score module configured to calculate a suitability score describing the suitability of the training data set to the inference data set as a function of the first and the second statistical data signatures; and an action module configured to perform an action related to the machine learning system in response to the suitability score satisfying an unsuitability threshold. ### Claim 2 (depends on 1) The apparatus of claim 1, wherein one or more of the training data set and the inference data set comprise feature data sets that are free of labels. ### Claim 3 (depends on 1) The apparatus of claim 1, wherein the score module calculates the suitability score on a per-feature basis based on the training data set and the inference data set. ### Claim 4 (depends on 3) The apparatus of claim 3, wherein: the training evaluation module is further configured to: determine a probability distribution of the training data set by assigning the values for a feature of the training data set to different groups; determine probabilities for each of the groups of the training data set, the probabilities indicating a likelihood of a value being in the training data set; and calculate an average training probability score for the training data set as a function of the probabilities for the groups; the inference evaluation module is further configured to: determine a probability distribution of the inference data set by assigning the values for a feature of the training data set to different groups that correspond to the groups of the training data set; determine probabilities for each of the groups of the inference data set, the probabilities indicating a likelihood of a value of the inference data set being in the training data set; and calculate an average inference probability score based on the probability distribution of the inference data set; and the score module is further configured to calculate the suitability score by normalizing the average inference probability score as a function of the average training probability score. ### Claim 5 (depends on 1) The apparatus of claim 1, wherein the score module calculates the suitability score across a plurality of features of the training data set and the inference data set. ### Claim 6 (depends on 5) The apparatus of claim 5, wherein the training and inference data sets comprise continuous feature data, and: the training evaluation module is further configured to: generate a Gaussian mixture model as a function of the training data set; determine a likelihood distribution of the training data set based on the generated Gaussian mixture model; and calculate an average training likelihood score based on the likelihood distribution of the training data set; the inference evaluation module is further configured to: determine a likelihood distribution of the inference data set based on the generated Gaussian mixture model; and calculate an average inference probability score based on the likelihood distribution of the inference data set; and the score module is further configured to calculate the suitability score by normalizing the average inference likelihood score as a function of the average training likelihood score. ### Claim 7 (depends on 5) The apparatus of claim 5, wherein the training and inference data sets comprise one or more of categorical and binary feature data, and: the training evaluation module is further configured to: generate a non-random forest of trees as a function of the training data set; determine a probability distribution of the training data set based on the generated non-random forest of trees; and calculate an average training probability score based on the probability distribution of the training data set; the inference evaluation module is further configured to: determine a probability distribution of the inference data set as a function of the non-random forest of trees generated based on the training data; and calculate an average inference probability score based on the probability distribution of the inference data set; and the score module is further configured to calculate the suitability score by normalizing the average inference probability score as a function of the average training probability score. ### Claim 8 (depends on 1) The apparatus of claim 1, wherein the inference data set is analyzed using a deep learning machine learning pipeline that comprises a plurality of learning layers, the inference data set selected from a learning layer of the plurality of learning layers that occurs previous to the final learning layer. ### Claim 9 (depends on 8) The apparatus of claim 8, wherein the inference data set is selected from the penultimate learning layer of the plurality of learning layers. ### Claim 10 (depends on 1) The apparatus of claim 1, wherein the action that the action module performs comprises changing the machine learning model to a different machine learning model that is more suitable for the inference data set based on the suitability score. ### Claim 11 (depends on 1) The apparatus of claim 1, wherein the action that the action module performs comprises retraining the machine learning model on a different training data set until the suitability score satisfies a suitability threshold. ### Claim 12 (depends on 1) The apparatus of claim 1, wherein the action that the action module performs comprises sending one or more of an alert, message, and notification that indicates the training data set is unsuitable for the inference data set. ### Claim 13 (depends on 1) The apparatus of claim 1, wherein the action that the action module performs comprises generating one or more labels for the features of the inference data set. ### Claim 14 (depends on 1) The apparatus of claim 1, wherein the machine learning system comprises a plurality of interconnected training and inference pipelines, the suitability score calculated in real-time and on an ongoing basis during machine learning processing to determine the suitability of the training set data to the inference data set at a particular pipeline. ### Claim 15 A method comprising: calculating a first statistical data signature for a training data set of a machine learning system using one or more predefined statistical algorithms, the training data set used to generate a machine learning model; calculating a second statistical data signature for an inference data set of the machine learning system using the one or more predefined statistical algorithms, the inference data set analyzed using the machine learning model; calculating a suitability score describing the suitability of the training data set to the inference data set as a function of the first and the second statistical data signatures; and performing an action related to the machine learning system in response to the suitability score satisfying an unsuitability threshold. ### Claim 16 (depends on 15) The method of claim 15, wherein one or more of the training data set and the inference data set comprise feature data sets that are free of labels. ### Claim 17 (depends on 15) The method of claim 15, further comprising: calculating the suitability score on a per-feature bases by: determining a probability distribution of the training data set by assigning the values for a feature of the training data set to different groups; determining probabilities for each of the groups of the training data set, the probabilities indicating a likelihood of a value being in the training data set; calculating an average training probability score for the training data set as a function of the probabilities for the groups; determining a probability distribution of the inference data set by assigning the values for a feature of the training data set to different groups that correspond to the groups of the training data set; determining probabilities for each of the groups of the inference data set, the probabilities indicating a likelihood of a value of the inference data set being in the training data set; calculating an average inference probability score based on the probability distribution of the inference data set; and calculating the suitability score by normalizing the average inference probability score as a function of the average training probability score. ### Claim 18 (depends on 15) The method of claim 15, further comprising: calculating the suitability score across a plurality of features of continuous feature data of the training data set and the inference data set by: generating a Gaussian mixture model as a function of the training data set; determining a likelihood distribution of the training data set based on the generated Gaussian mixture model; calculating an average training likelihood score based on the likelihood distribution of the training data set; determining a likelihood distribution of the inference data set based on the generated Gaussian mixture model; calculating an average inference likelihood score based on the likelihood distribution of the inference data set; and calculating the suitability score by normalizing the average inference likelihood score as a function of the average training likelihood score. ### Claim 19 (depends on 15) The method of claim 15, further comprising: calculating the suitability score across a plurality of features of one or more of categorical and binary feature data of the training data set and the inference data set by: generating a non-random forest of trees as a function of the training data set; determining a probability distribution of the training data set based on the generated non-random forest of trees; calculating an average training probability score based on the probability distribution of the training data set; determining a probability distribution of the inference data set as a function of the non-random forest of trees generated based on the training data set; calculating an average inference probability score based on the probability distribution of the inference data set; and calculating the suitability score by normalizing the average inference probability score as a function of the average training probability score. ### Claim 20 An apparatus comprising: means for calculating a first statistical data signature for a training data set of a machine learning system using one or more predefined statistical algorithms, the training data set used to generate a machine learning model; means for calculating a second statistical data signature for an inference data set of the machine learning system using the one or more predefined statistical algorithms, the inference data set analyzed using the machine learning model; means for calculating a suitability score describing the suitability of the training data set to the inference data set as a function of the first and the second statistical data signatures; and means for performing an action related to the machine learning system in response to the suitability score satisfying an unsuitability threshold.