# US 2021/0241047 A1 · Determining rationale for a prediction of a machine learning based model > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0241047 A1 - Document ID: US20210241047A1 - Status: Published application - Kind: A1 - Application number: 16/778,925 - Filed: 2020-01-31 - Published: 2021-08-05 - Earliest claimed: 2020-01-31 - Assignees: salesforce.com, inc. - Applicants: salesforce.com, inc. - Inventors: Rakesh Ganapathi Karanth; Arun Kumar Jagota; Kaushal Bansal; Amrita Dasgupta - Web page: https://patentgrep.com/patent/US20210241047A1 ## CPC - G06N 20/20: Ensemble learning - G06F 18/2134: based on separation criteria, e.g. independent component analysis - G06F 18/24137: {Distances to cluster centroïds} - G06F 18/24323: {Tree-organised classifiers} - G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound - G06N 5/045: Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence - G06N 20/00: Machine learning - G06N 3/08: Learning methods - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract An online system performs predictions for real-time tasks and near real-time tasks that need to be performed by a deadline. A client device receives a real-time machine learning based model associated with a measure of accuracy. If the client device determines that a task can be performed using predictions having less than the specified measure of accuracy, the client device uses the real-time machine learning based model. If the client device determines that a higher level of accuracy of results is required, the client device sends a request to an online system. The online system provides a prediction along with a string representing a rationale for the prediction. ## Claims ### Claim 1 A computer implemented method for providing rationale for machine learning based predictions, the method comprising: receiving, by an online system, a request for making a prediction using a decision tree based machine learning model, the request comprising a feature vector for providing as input to the machine learning model; providing the feature vector as input to the decision tree based machine learning model; executing the decision tree based machine learning model to generate a prediction; obtaining a trace of execution of the decision tree based machine learning model, the trace identifying an execution path taken through the decision tree during execution of the decision tree based machine learning model; selecting one or more features in the trace of execution of the decision tree based machine learning model; generating a string based on information describing the selected one or more features, the string indicating a rationale for the prediction of the decision tree based machine learning model; and sending the generated string to a client device for presentation via a user interface. ### Claim 2 (depends on 1) The method of claim 1, further comprising: determining a type of contribution of each of the selected one or more features to the prediction, the type of contribution representing one of a positive contribution or a negative contribution; and wherein the generated string includes information describing the type of contribution of each of the selected one or more features. ### Claim 3 (depends on 1) The method of claim 1, further comprising: determining a value indicating an amount of contribution of each of the selected one or more features to the prediction; and wherein the generated string includes information describing the amount of contribution of each of the selected one or more features. ### Claim 4 (depends on 3) The method of claim 3, wherein determining the value indicating the amount of contribution of each of the selected one or more features to the prediction comprises: for each of the selected one or more features: determining a prediction based on the feature in isolation; determining the value indicating the amount of contribution to the prediction based on the feature in isolation. ### Claim 5 (depends on 4) The method of claim 4, further comprising: determining a ground truth mean representing an aggregate of values of the feature in a dataset; and wherein the value indicating the amount of contribution is determined based on a difference between the prediction based on the feature in isolation and the ground truth mean. ### Claim 6 (depends on 1) The method of claim 1, wherein the generated string identifies features based on their significance in determining the prediction, the significance determined based on the order in which the feature occurs in the execution path. ### Claim 7 (depends on 1) The method of claim 1, wherein the online system is a multi-tenant system storing data for a plurality of tenants, wherein the machine learning model is trained for a particular tenant from the plurality of tenants. ### Claim 8 (depends on 1) The method of claim 1, wherein the decision tree based machine learning model is configured to determine a measure indicating a value of one or more items, wherein the online system stores database records comprising attributes describing the one or more items. ### Claim 9 A non-transitory computer readable storage medium storing instructions that when executed by a processor cause the processor to execute steps for providing rationale for machine learning based predictions, the steps comprising: receiving, by an online system, a request for making a prediction using a decision tree based machine learning model, the request comprising a feature vector for providing as input to the machine learning model; providing the feature vector as input to the decision tree based machine learning model; executing the decision tree based machine learning model to generate a prediction; obtaining a trace of execution of the decision tree based machine learning model, the trace identifying an execution path taken through the decision tree during execution of the decision tree based machine learning model; selecting one or more features in the trace of execution of the decision tree based machine learning model; generating a string based on information describing the selected one or more features, the string indicating a rationale for the prediction of the decision tree based machine learning model; and sending the generated string to a client device for presentation via a user interface. ### Claim 10 (depends on 9) The non-transitory computer readable storage medium of claim 9, wherein the instructions further cause the processor to perform steps comprising: determining a type of contribution of each of the selected one or more features to the prediction, the type of contribution representing one of a positive contribution or a negative contribution; and wherein the generated string includes information describing the type of contribution of each of the selected one or more features. ### Claim 11 (depends on 9) The non-transitory computer readable storage medium of claim 9, wherein the instructions further cause the processor to perform steps comprising: determining a value indicating an amount of contribution of each of the selected one or more features to the prediction; and wherein the generated string includes information describing the amount of contribution of each of the selected one or more features. ### Claim 12 (depends on 11) The non-transitory computer readable storage medium of claim 11, wherein determining the value indicating the amount of contribution of each of the selected one or more features to the prediction causes the processor to perform steps comprising: for each of the selected one or more features: determining a prediction based on the feature in isolation; determining the value indicating the amount of contribution to the prediction based on the feature in isolation. ### Claim 13 (depends on 12) The non-transitory computer readable storage medium of claim 12, wherein the instructions further cause the processor to perform steps comprising: determining a ground truth mean representing an aggregate of values of the feature in a dataset; and wherein the value indicating the amount of contribution is determined based on a difference between the prediction based on the feature in isolation and the ground truth mean. ### Claim 14 (depends on 9) The non-transitory computer readable storage medium of claim 9, wherein the generated string identifies features based on their significance in determining the prediction, the significance determined based on the order in which the feature occurs in the execution path. ### Claim 15 A computer system comprising: a processor; and a non-transitory computer readable storage medium storing instructions, that when executed by a processor cause the processor to execute steps for providing rationale for machine learning based predictions, the steps comprising: receiving, by an online system, a request for making a prediction using a decision tree based machine learning model, the request comprising a feature vector for providing as input to the machine learning model; providing the feature vector as input to the decision tree based machine learning model; executing the decision tree based machine learning model to generate a prediction; obtaining a trace of execution of the decision tree based machine learning model, the trace identifying an execution path taken through the decision tree during execution of the decision tree based machine learning model; selecting one or more features in the trace of execution of the decision tree based machine learning model; generating a string based on information describing the selected one or more features, the string indicating a rationale for the prediction of the decision tree based machine learning model; and sending the generated string to a client device for presentation via a user interface. ### Claim 16 (depends on 15) The computer system of claim 15, wherein the instructions further cause the processor to perform steps comprising: determining a type of contribution of each of the selected one or more features to the prediction, the type of contribution representing one of a positive contribution or a negative contribution; and wherein the generated string includes information describing the type of contribution of each of the selected one or more features. ### Claim 17 (depends on 15) The computer system of claim 15, wherein the instructions further cause the processor to perform steps comprising: determining a value indicating an amount of contribution of each of the selected one or more features to the prediction; and wherein the generated string includes information describing the amount of contribution of each of the selected one or more features. ### Claim 18 (depends on 17) The computer system of claim 17, wherein determining the value indicating the amount of contribution of each of the selected one or more features to the prediction causes the processor to perform steps comprising: for each of the selected one or more features: determining a prediction based on the feature in isolation; determining the value indicating the amount of contribution to the prediction based on the feature in isolation. ### Claim 19 (depends on 18) The computer system of claim 18, wherein the instructions further cause the processor to perform steps comprising: determining a ground truth mean representing an aggregate of values of the feature in a dataset; and wherein the value indicating the amount of contribution is determined based on a difference between the prediction based on the feature in isolation and the ground truth mean. ### Claim 20 (depends on 15) The computer system of claim 15, wherein the generated string identifies features based on their significance in determining the prediction, the significance determined based on the order in which the feature occurs in the execution path.