# US 10,311,368 B2 · Analytic system for graphical interpretability of and improvement of machine learning models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,311,368 B2 - Document ID: US10311368B2 - Status: Granted - Kind: B2 - Application number: 15/928,363 - Filed: 2018-03-22 - Published: 2019-03-14 - Granted: 2019-06-04 - Earliest claimed: 2017-09-12 - Assignees: SAS Institute Inc. - Applicants: SAS Institute Inc. - Inventors: Namita Dilip Lokare; Jorge Manuel Gomes da Silva; Ilknur Kaynar Kabul; Gregory Naisat - Web page: https://patentgrep.com/patent/US10311368B2 ## CPC - G06N 3/08: Learning methods - G06N 20/00: Machine learning - G06N 3/0499: Feedforward networks - G06N 3/09: Supervised learning - G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06N 20/10: using kernel methods, e.g. support vector machines [SVM] - G06N 20/20: Ensemble learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A computing device provides a cluster connectivity graph presented on a display to summarize machine learning model performance. A classification value is predicted is predicted for a response variable value of each observation vector using a trained model. Observation vectors are divided into overlapping data slices that are separately clustered using the predicted classification value to define a set of clusters. A number of observations in each cluster is computed. An accuracy measure is computed for each cluster based on the predicted classification value. A number of overlapping observations between each pair of clusters is computed. The cluster connectivity graph includes a node for each cluster. A size of each node is determined from the computed number of observations. A fill-pattern of each node is determined from the computed accuracy measure. A connector line between each pair of nodes is determined from the computed number of overlapping observations. ## Claims ### Claim 1 A non-transitory computer-readable medium having stored thereon computer-readable instructions that when executed by a computing device cause the computing device to: read a dataset that includes a plurality of observation vectors, wherein each observation vector of the plurality of observation vectors includes an explanatory variable value for an explanatory variable and a response variable value for a response variable; train a model using the plurality of observation vectors; predict a classification value for the response variable for each observation vector of the plurality of observation vectors using a respective explanatory variable value input to the trained model; divide the plurality of observation vectors into a plurality of overlapping data slices; separately cluster each data slice of the plurality of overlapping data slices using the predicted classification value to define a set of clusters for the plurality of observation vectors; compute a number of observations in each cluster of the set of clusters; compute an accuracy measure for each cluster of the set of clusters based on a comparison between the predicted classification value and a respective response variable value of each observation vector of the plurality of observation vectors; compute a number of overlapping observations between each pair of clusters of the set of clusters; and present a cluster connectivity graph on a display, wherein the cluster connectivity graph includes a node for each cluster of the set of clusters, wherein a size of each node is determined from the computed number of observations, wherein a fill-pattern of each node is determined from the computed accuracy measure, wherein a connector line between each pair of nodes is determined from the computed number of overlapping observations. ### Claim 2 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the model is a machine learning model selected from the group consisting of a decision tree model, a factorization machine model, a forest model, a gradient boosting tree model, a neural network model, and a support vector machine model. ### Claim 3 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the classification value is predicted based on a probability value output from the trained model. ### Claim 4 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the classification value is predicted based on a probability value output from a layer of the trained model. ### Claim 5 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein dividing the plurality of observation vectors comprises: sorting the plurality of observation vectors in order based on the response variable value; and splitting the sorted plurality of observation vectors into a predefined number of data slices based on a predefined percentage overlap between data slices. ### Claim 6 (depends on 5) The non-transitory computer-readable medium of claim 5, wherein the predefined percentage overlap is greater than zero. ### Claim 7 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the accuracy measure is computed based on a predefined objective function that provides a measure of model error. ### Claim 8 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the computed accuracy measure is a ratio relative to an overall accuracy measure. ### Claim 9 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein a width is defined for each connector line, wherein the width is greater than or equal to zero. ### Claim 10 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the fill-pattern is a color selected based on the computed accuracy measure for a respective cluster of the node. ### Claim 11 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the computer-readable instructions further cause the computing device to: determine a poor performing cluster of the set of clusters based on the computed accuracy measure; select a subset of observation vectors included in the determined poor performing cluster; compare the explanatory variable value of the selected subset of observation vectors to the explanatory variable value of the plurality of observation vectors; determine a model improvement based on the comparison, wherein the model improvement splits the plurality of observation vectors into a first set of observation vectors and a second set of observation vectors based on a split value of the explanatory variable; train a first model using the first set of observation vectors; train a second model using the second set of observation vectors; and output the trained first model, the trained second model, and the split value of the explanatory variable. ### Claim 12 (depends on 11) The non-transitory computer-readable medium of claim 11, wherein the poor performing cluster is determined automatically when the computed accuracy measure is less than a threshold accuracy measure. ### Claim 13 (depends on 11) The non-transitory computer-readable medium of claim 11, wherein the poor performing cluster is determined as a result of an interaction by a user with the presented cluster connectivity graph. ### Claim 14 (depends on 11) The non-transitory computer-readable medium of claim 11, wherein the split value is determined automatically. ### Claim 15 (depends on 11) The non-transitory computer-readable medium of claim 11, wherein the split value is determined based on a presented comparison. ### Claim 16 A computing device comprising: a processor; and a non-transitory computer-readable medium operably coupled to the processor, the computer-readable medium having computer-readable instructions stored thereon that, when executed by the processor, cause the computing device to read a dataset that includes a plurality of observation vectors, wherein each observation vector of the plurality of observation vectors includes an explanatory variable value for an explanatory variable and a response variable value for a response variable; train a model using the plurality of observation vectors; predict a classification value for the response variable for each observation vector of the plurality of observation vectors using a respective explanatory variable value input to the trained model; divide the plurality of observation vectors into a plurality of overlapping data slices; separately cluster each data slice of the plurality of overlapping data slices using the predicted classification value to define a set of clusters for the plurality of observation vectors; compute a number of observations in each cluster of the set of clusters; compute an accuracy measure for each cluster of the set of clusters based on a comparison between the predicted classification value and a respective response variable value of each observation vector of the plurality of observation vectors; compute a number of overlapping observations between each pair of clusters of the set of clusters; and present a cluster connectivity graph on a display, wherein the cluster connectivity graph includes a node for each cluster of the set of clusters, wherein a size of each node is determined from the computed number of observations, wherein a fill-pattern of each node is determined from the computed accuracy measure, wherein a connector line between each pair of nodes is determined from the computed number of overlapping observations. ### Claim 17 A method provides a visualization that summarizes a performance of a machine learning model to support model improvement and understanding, the method comprising: reading, by a computing device, a dataset that includes a plurality of observation vectors, wherein each observation vector of the plurality of observation vectors includes an explanatory variable value for an explanatory variable and a response variable value for a response variable; training, by the computing device, a model using the plurality of observation vectors; predicting, by the computing device, a classification value for the response variable for each observation vector of the plurality of observation vectors using a respective explanatory variable value input to the trained model; dividing, by the computing device, the plurality of observation vectors into a plurality of overlapping data slices; separately clustering, by the computing device, each data slice of the plurality of overlapping data slices using the predicted classification value to define a set of clusters for the plurality of observation vectors; computing, by the computing device, a number of observations in each cluster of the set of clusters; computing, by the computing device, an accuracy measure for each cluster of the set of clusters based on a comparison between the predicted classification value and a respective response variable value of each observation vector of the plurality of observation vectors; computing, by the computing device, a number of overlapping observations between each pair of clusters of the set of clusters; and presenting, by the computing device, a cluster connectivity graph on a display, wherein the cluster connectivity graph includes a node for each cluster of the set of clusters, wherein a size of each node is determined from the computed number of observations, wherein a fill-pattern of each node is determined from the computed accuracy measure, wherein a connector line between each pair of nodes is determined from the computed number of overlapping observations. ### Claim 18 (depends on 17) The method of claim 17, wherein the model is a machine learning model selected from the group consisting of a decision tree model, a factorization machine model, a forest model, a gradient boosting tree model, a neural network model, and a support vector machine model. ### Claim 19 (depends on 17) The method of claim 17, wherein the classification value is predicted based on a probability value output from the trained model. ### Claim 20 (depends on 17) The method of claim 17, wherein the classification value is predicted based on a probability value output from a layer of the trained model. ### Claim 21 (depends on 17) The method of claim 17, wherein dividing the plurality of observation vectors comprises: sorting, by the computing device, the plurality of observation vectors in order based on the response variable value; and splitting, by the computing device, the sorted plurality of observation vectors into a predefined number of data slices based on a predefined percentage overlap between data slices. ### Claim 22 (depends on 21) The method of claim 21, wherein the predefined percentage overlap is greater than zero. ### Claim 23 (depends on 17) The method of claim 17, wherein the accuracy measure is computed based on a predefined objective function that provides a measure of model error. ### Claim 24 (depends on 17) The method of claim 17, wherein the computed accuracy measure is a ratio relative to an overall accuracy measure. ### Claim 25 (depends on 17) The method of claim 17, wherein a width is defined for each connector line, wherein the width is greater than or equal to zero. ### Claim 26 (depends on 17) The method of claim 17, wherein the fill-pattern is a color selected based on the computed accuracy measure for a respective cluster of the node. ### Claim 27 (depends on 17) The method of claim 17, further comprising: determining, by the computing device, a poor performing cluster of the set of clusters based on the computed accuracy measure; selecting, by the computing device, a subset of observation vectors included in the determined poor performing cluster; comparing, by the computing device, the explanatory variable value of the selected subset of observation vectors to the explanatory variable value of the plurality of observation vectors; determining, by the computing device, a model improvement based on the comparison, wherein the model improvement splits the plurality of observation vectors into a first set of observation vectors and a second set of observation vectors based on a split value of the explanatory variable; training, by the computing device, a first model using the first set of observation vectors; training, by the computing device, a second model using the second set of observation vectors; and outputting, by the computing device, the trained first model, the trained second model, and the split value of the explanatory variable. ### Claim 28 (depends on 27) The method of claim 27, wherein the poor performing cluster is determined as a result of an interaction by a user with the presented cluster connectivity graph. ### Claim 29 (depends on 27) The method of claim 27, wherein the poor performing cluster is determined automatically when the computed accuracy measure is less than a threshold accuracy measure. ### Claim 30 (depends on 27) The method of claim 27, wherein the split value is determined automatically.