# US 12,118,447 B2 · Model interpretation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,447 B2 - Document ID: US12118447B2 - Status: Granted - Kind: B2 - Application number: 18/137,972 - Filed: 2023-04-21 - Published: 2024-01-04 - Granted: 2024-10-15 - Earliest claimed: 2018-04-20 - Assignees: H2O.ai Inc. - Applicants: H2O.ai Inc. - Inventors: Mark Chan; Navdeep Gill; Patrick Hall - Web page: https://patentgrep.com/patent/US12118447B2 ## CPC - G06N 20/20: Ensemble learning - G06F 18/2115: by evaluating different subsets according to an optimisation criterion, e.g. class separability, forward selection or backward elimination - G06F 18/23213: with fixed number of clusters, e.g. K-means clustering - G06F 18/24323: {Tree-organised classifiers} - G06N 3/09: Supervised learning - G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06N 20/00: Machine learning - G06N 3/045: Combinations of networks - G06N 20/10: using kernel methods, e.g. support vector machines [SVM] - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract An indication of a selection of an entry associated with a machine learning model is received. One or more interpretation views associated with one or more machine learning models are dynamically updated based on the selected entry. ## Claims ### Claim 1 A method, comprising: receiving an indication of a selection of an entry associated with a linear surrogate machine learning model, wherein the linear surrogate machine learning model approximates an output associated with a machine learning model; in response to the received indication of the selection of the entry associated with the linear surrogate machine learning model, dynamically updating one or more interpretation views associated with one or more other surrogate machine learning models based on the selected entry, wherein dynamically updating one or more interpretation views associated with the one or more other surrogate machine learning models includes updating a first non-linear surrogate machine learning model comprising a feature importance model that includes corresponding global feature importance values and corresponding local importance feature values for a plurality of features; determining whether a local feature importance value associated with the selected entry correlates with a global feature importance value associated with the selected entry; flagging the selected entry in response to determining that the local feature importance value associated with the selected entry does not correlate with the global feature importance value associated with the selected entry, wherein the local feature importance value associated with the selected entry does not correlate with the global feature importance value associated with the selected entry in the event a difference between a first feature local feature importance value of a first feature of the plurality of features and a first feature global feature importance value of the first feature is greater than a threshold value; and in response to a threshold number of entries associated with the linear surrogate machine learning model being flagged, retraining the first non-linear surrogate machine learning model that includes the feature importance model. ### Claim 2 (depends on 1) The method of claim 1, wherein a corresponding global feature importance value for the first feature of the plurality of features indicates an importance of the first feature for a global surrogate model. ### Claim 3 (depends on 2) The method of claim 2, wherein the corresponding global feature importance value for the first feature indicates a number of times in a decision tree ensemble that the first feature was selected to split decision tree of the decision tree ensemble. ### Claim 4 (depends on 2) The method of claim 2, wherein the corresponding global feature importance value for the first feature is scaled to a number between 0 and 1. ### Claim 5 (depends on 2) The method of claim 2, wherein the corresponding global feature importance value for the first feature is weighted based on a location of the first feature in a decision tree. ### Claim 6 (depends on 1) The method of claim 1, wherein a corresponding local feature importance value for the first feature of the plurality of features indicates an importance of the first feature for the selected entry. ### Claim 7 (depends on 6) The method of claim 6, wherein the corresponding local feature importance value for the first feature is determined by computing a leave-one-covariate-out (LOCO) value for the first feature. ### Claim 8 (depends on 1) The method of claim 1, further comprising providing the first non-linear surrogate machine learning model at least in part by: determining the corresponding global feature importance values for the plurality of features; and determining the corresponding local importance feature values for the plurality of features. ### Claim 9 (depends on 1) The method of claim 1, further comprising ranking the plurality of features. ### Claim 10 (depends on 9) The method of claim 9, wherein the plurality of features is ranked based on their corresponding global feature importance values. ### Claim 11 (depends on 9) The method of claim 9, wherein the plurality of features is ranked based on their corresponding local feature importance values. ### Claim 12 (depends on 9) The method of claim 9, further comprising providing a visualization of the ranking of the plurality of features. ### Claim 13 (depends on 12) The method of claim 12, wherein the visualization includes a top number of the plurality of features. ### Claim 14 (depends on 12) The method of claim 12, wherein the visualization includes a top percentage of the plurality of features. ### Claim 15 A system, comprising: a processor configured to: receive an indication of a selection of an entry associated with a linear surrogate machine learning model, wherein the linear surrogate machine learning model approximates an output associated with a machine learning model; in response to the received indication of the selection of the entry associated with the linear surrogate machine learning model, dynamically update one or more interpretation views associated with one or more other surrogate machine learning models based on the selected entry, wherein to dynamically update one or more interpretation views associated with the one or more other surrogate machine learning models, the processor is configured to update a first non-linear surrogate machine learning model comprising a feature importance model that includes corresponding global feature importance values and corresponding local importance feature values for a plurality of features; determine whether a local feature importance value associated with the selected entry correlates with a global feature importance value associated with the selected entry; flag the selected entry in response to determining that the local feature importance value associated with the selected entry does not correlate with the global feature importance value associated with the selected entry, wherein the local feature importance value associated with the selected entry does not correlate with the global feature importance value associated with the selected entry in the event a difference between a first feature local feature importance value of a first feature of the plurality of features and a first feature global feature importance value of the first feature is greater than a threshold value; and in response to a threshold number of entries associated with the linear surrogate machine learning model being flagged, retrain the first non-linear surrogate machine learning model that includes the feature importance model; and a memory coupled to the processor and configured to provide the processor with instructions. ### Claim 16 (depends on 15) The system of claim 15, wherein a global feature importance value for the first feature of the plurality of features indicates an importance of the first feature for a global surrogate model. ### Claim 17 (depends on 15) The system of claim 15, wherein a local feature importance value for the first feature of the plurality of features indicates an importance of the first feature for the selected entry. ### Claim 18 (depends on 15) The system of claim 15, wherein the processor is configured to rank the plurality of features. ### Claim 19 (depends on 18) The system of claim 18, wherein the processor is further configured to provide a visualization of the ranking of the plurality of features. ### Claim 20 A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for: receiving an indication of a selection of an entry associated with a linear surrogate machine learning model, wherein the linear surrogate machine learning model approximates an output associated with a machine learning model; in response to the received indication of the selection of the entry associated with the linear surrogate machine learning model, dynamically updating one or more interpretation views associated with one or more other surrogate machine learning models based on the selected entry, wherein dynamically updating one or more interpretation views associated with the one or more other surrogate machine learning models includes updating a first non-linear surrogate machine learning model comprising a feature importance model that includes corresponding global feature importance values and corresponding local importance feature values for a plurality of features; determining whether a local feature importance value associated with the selected entry correlates with a global feature importance value associated with the selected entry; flagging the selected entry in response to determining that the local feature importance value associated with the selected entry does not correlate with the global feature importance value associated with the selected entry, wherein the local feature importance value associated with the selected entry does not correlate with the global feature importance value associated with the selected entry in the event a difference between a first feature local feature importance value of a first feature of the plurality of features and a first feature global feature importance value of the first feature is greater than a threshold value; and in response to a threshold number of entries associated with the linear surrogate machine learning model being flagged, retraining the first non-linear surrogate machine learning model that includes the feature importance model.