# US 2021/0241033 A1 · Test suite for different kinds of biases in data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0241033 A1 - Document ID: US20210241033A1 - Status: Published application - Kind: A1 - Application number: 16/777,912 - Filed: 2020-01-31 - Published: 2021-08-05 - Earliest claimed: 2020-01-31 - Assignees: salesforce.com, inc. - Applicants: salesforce.com, inc. - Inventors: Michael YANG - Web page: https://patentgrep.com/patent/US20210241033A1 ## CPC - G06N 20/00: Machine learning - G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade} - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06F 18/23: Clustering techniques - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract There is provided computer implemented method for detecting and reducing or removing bias for generating a machine learning model, comprising: prior to generating the machine learning model: receiving a training dataset, comprising target inputs, each comprising parameters and labelled with a corresponding target output, wherein at least one of the parameters of at least of the target inputs comprises a sensitive parameter indicative of the corresponding target input assigned to a sensitive group that is potentially biased against other target inputs that are excluded from the sensitive group, analyzing the training dataset to identify target inputs affected by label bias when a statistically significant difference is detected between target inputs assigned to the sensitive group and target inputs excluded from the sensitive group, correcting labels of the target inputs affected by label bias, and generating the machine learning model using the corrected labels. ## Claims ### Claim 1 A computer implemented method for detecting and reducing or removing bias for generating a machine learning model, comprising: (i) prior to generating the machine learning model: receiving a training dataset, comprising a plurality of target inputs, each comprising a plurality of parameters and labelled with a corresponding target output, wherein at least one of the plurality of parameters of at least of the plurality of target inputs comprises a sensitive parameter indicative of the corresponding target input assigned to a sensitive group that is potentially biased against other target inputs that are excluded from the sensitive group; analyzing the training dataset to identify target inputs affected by label bias when a statistically significant difference is detected between target inputs assigned to the sensitive group and target inputs excluded from the sensitive group; correcting labels of the target inputs affected by label bias; and (ii) generating the machine learning model using the corrected labels. ### Claim 2 (depends on 1) The method of claim 1, further comprising: computing by a score computing machine learning model, for each respective target input, a probability of the respective target input being assigned to the sensitive group according to a respective value of the corresponding sensitive parameter. ### Claim 3 (depends on 2) The method of claim 2, further comprising: cluster the plurality of target inputs into a plurality of clusters according to corresponding computed probabilities, wherein for each respective cluster, target inputs assigned to the respective cluster are associated with probabilities within a certain probability value range, wherein each cluster includes target inputs assigned to the sensitive group and target inputs excluded from the sensitive group, for each respective cluster: determining whether the statistically significant difference exists between target inputs assigned to the sensitive group and other target inputs excluded from the sensitive group; and identifying label bias for the respective cluster when the statistically significant difference is detected, wherein the at least one target input affected by label bias comprises the target inputs of the respective cluster, including the target inputs assigned to the sensitive group and target inputs excluded from the sensitive group. ### Claim 4 (depends on 3) The method of claim 3, wherein correcting labels of the target inputs affected by label bias comprises correcting labels for the target inputs assigned to the sensitive groups and labels of the other target inputs excluded from the sensitive group. ### Claim 5 (depends on 4) The method of claim 4, wherein the correcting labels comprises assigning the same label to all of the target inputs assigned to the sensitive groups and to all of the other target inputs excluded from the sensitive group. ### Claim 6 (depends on 2) The method of claim 2, wherein the probability of the respective target input being assigned to the sensitive group is computed by the score computing machine learning model performing a causal inference process, wherein a treatment of the causal inference procedure is the sensitive parameter. ### Claim 7 (depends on 6) The method of claim 6, wherein the causal inference process comprises a propensity score matching (PSM) process, and the probability denotes the propensity score. ### Claim 8 (depends on 2) The method of claim 2, further comprising: computing accuracy of the score computing machine learning model for computing the probability of the respective target input being assigned to the sensitive group; and identifying sampling bias between target inputs of the sensitive group and other target inputs excluded from the sensitive group when the accuracy of the score computing machine learning model is above a threshold. ### Claim 9 (depends on 8) The method of claim 8, wherein (ii) generating further comprises generating one respective machine learning model for the target inputs of the sensitive group and generating another respective machine learning model for the other target inputs excluded from the sensitive group. ### Claim 10 (depends on 1) The method of claim 1, wherein each of the plurality of target inputs comprises the sensitive parameter, wherein the target inputs assigned to the sensitive group include a value of the sensitive parameter meeting a requirement, and other target inputs excluded from the sensitive group include another value of the sensitive parameter that does not meet the requirement. ### Claim 11 (depends on 1) The method of claim 1, wherein the sensitive parameter is selected from the group consisting of: gender, race, and age. ### Claim 12 A computer implemented method for detecting and reducing or removing bias for generating a machine learning model, comprising: (i) prior to generating the machine learning model: receiving a training dataset, comprising a plurality of target inputs, each comprising a plurality of parameters and labelled with a corresponding target output, wherein at least one of the plurality of parameters of at least of the plurality of target inputs comprises a sensitive parameter indicative of the corresponding target input assigned to a sensitive group that is potentially biased against other target inputs that are excluded from the sensitive group; analyzing the training dataset to detect sampling bias between the target inputs of the sensitive group and other target inputs excluded from the sensitive group; and (ii) generating one respective machine learning model for the target inputs of the sensitive group and generating another respective machine learning model for the other target inputs excluded from the sensitive group. ### Claim 13 (depends on 12) The method of claim 12, further comprising computing accuracy of a score computing machine learning model that computes a probability of a certain target input being assigned to the sensitive group; and detecting the sampling bias when the accuracy of the score computing machine learning model is above a threshold. ### Claim 14 A system for detecting and reducing or removing bias for generating a machine learning model, comprising: at least one hardware processor executing a code for: (i) prior to generating the machine learning model: receiving a training dataset, comprising a plurality of target inputs, each comprising a plurality of parameters and labelled with a corresponding target output, wherein at least one of the plurality of parameters of at least of the plurality of target inputs comprises a sensitive parameter indicative of the corresponding target input assigned to a sensitive group that is potentially biased against other target inputs that are excluded from the sensitive group; analyzing the training dataset to identify target inputs affected by label bias when a statistically significant difference is detected between target inputs assigned to the sensitive group and target inputs excluded from the sensitive group; correcting labels of the target inputs affected by label bias; and (ii) generating the machine learning model using the corrected labels. ### Claim 15 (depends on 14) The system of claim 14, further comprising, code for: computing accuracy of a score computing machine learning model for computing the probability of the respective target input being assigned to the sensitive group; and identifying sampling bias between target inputs of the sensitive group and other target inputs excluded from the sensitive group when the accuracy of the score computing machine learning model is above a threshold. ### Claim 16 (depends on 15) The system of claim 15, wherein (ii) generating further comprises generating one respective machine learning model for the target inputs of the sensitive group and generating another respective machine learning model for the other target inputs excluded from the sensitive group.