# US 2022/0012610 A1 · Methods for detecting and monitoring bias in a software application using artificial intelligence and devices thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0012610 A1
- Document ID: US20220012610A1
- Status: Published application
- Kind: A1
- Application number: 16/927,501
- Filed: 2020-07-13
- Published: 2022-01-13
- Earliest claimed: 2020-07-13
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Kazuki Sekiguchi; Fumihiko Terui; Pinaki Chandra Dey
- Web page: https://patentgrep.com/patent/US20220012610A1

## CPC

- G06N 5/04: Inference or reasoning models
- G06F 11/302: {where the computing system component is a software system}
- G06F 11/34: Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation {; Recording or statistical evaluation of user activity, e.g. usability assessment}
- G06N 20/00: Machine learning
- G06Q 40/03: Credit; Loans; Processing thereof
- G06Q 40/08: Insurance
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

A computer-implemented method for detecting and monitoring bias in an application includes index training data and obtaining a plurality of correlation values of one or more features in the indexed training data with a target variable. For each of the one or more features, a first value and a favorable result and a second value along with the unfavorable result is calculated. An absolute value of a difference between the calculated first value and the calculated second value is calculated. A total sum of the calculated absolute value of the plurality of correlation values of the one of the one or more features is calculated.

## Claims

### Claim 1

A method for monitoring and detecting bias in an application, the method comprising:
indexing, by a bias monitoring computing system, training data;
obtaining, by the bias monitoring computing system, a plurality of correlation values of one or more features in the indexed training data with a target variable; and
for each of the one or more features:
calculating, by the bias monitoring computing system, a first value of correlation of the plurality of correlation values of the one of the one or more features and a favorable result of a judgement by a trained artificial intelligence model;
calculating, by the bias monitoring computing system, a second value of correlation of the plurality of correlation values of the one of the one or more features and a unfavorable result of a judgement by the trained artificial intelligence model;
calculating, by the bias monitoring computing system, an absolute value of a difference between the calculated first value and the calculated second value;
calculating, by the bias monitoring computing system, a total sum of the calculated absolute value of the plurality of correlation values of the one of the one or more features; and
identifying and monitoring, by the bias monitoring computing system, a bias in the one or more features using the calculated first value, the second value, the absolute value, and the total sum.

### Claim 2 (depends on 1)

The method as set forth in claim 1, further comprising: indexing, by the bias monitoring computing system, feedback data received from a client device.

### Claim 3 (depends on 1)

The method as set forth in claim 1, further comprising, obtaining, by the bias monitoring computing system, training data from a training data server.

### Claim 4 (depends on 2)

The method as set forth in claim 2, further comprising, calculating, by the bias monitoring computing system, a second total sum of the calculated absolute value of the plurality of correlation values of the one of the one or more features for the feedback data.

### Claim 5 (depends on 4)

The method as set forth in claim 4, further comprising, presenting, by the bias monitoring computing system, a correlation change value determined by a difference between the calculated total sum of the calculated absolute value associated with the training data and the calculated total sum of the calculated absolute value associated with the feedback data.

### Claim 6 (depends on 5)

The method as set forth in claim 5, wherein the correlation change value represents a candidate of a feature to be monitored.

### Claim 7

A non-transitory machine readable medium having stored thereon instructions comprising machine executable code which when executed by at least one machine causes the machine to:
index training data;
obtain a plurality of correlation values of one or more features in the indexed training data with a target variable; and
for each of the one or more features:
calculate a first value of correlation of the plurality of correlation values of the one of the one or more features and a favorable result of a judgement by a trained artificial intelligence model;
calculate a second value of correlation of the plurality of correlation values of the one of the one or more features and a unfavorable result of a judgement by the trained artificial intelligence model;
calculate an absolute value of a difference between the calculated first value and the calculated second value;
calculate a total sum of the calculated absolute value of the plurality of correlation values of the one of the one or more features; and
identify and monitor a bias in the one or more features using the calculated first value, the second value, the absolute value, and the total sum.

### Claim 8 (depends on 7)

The medium as set forth in claim 7, further comprising, indexing feedback data received from a client device.

### Claim 9 (depends on 7)

The medium as set forth in claim 7, further comprising, obtaining training data from a training data server.

### Claim 10 (depends on 7)

The medium as set forth in claim 7, further comprising, calculating a second total sum of the calculated absolute value of the plurality of correlation values of the one of the one or more features for the feedback data.

### Claim 11 (depends on 10)

The medium as set forth in claim 10, further comprising, presenting a correlation change value determined by a difference between the calculated total sum of the calculated absolute value associated with the training data and the calculated total sum of the calculated absolute value associated with the feedback data.

### Claim 12 (depends on 11)

The medium as set forth in claim 11, wherein the correlation change value represents a candidate of a feature to be monitored.

### Claim 13

A bias monitoring computing system, comprising a memory comprising programmed instructions stored in the memory and one or more processors configured to be capable of executing the programmed instructions stored in the memory to:
index training data;
obtain a plurality of correlation values of one or more features in the indexed training data with a target variable; and
for each of the one or more features:
calculate a first value of correlation of the plurality of correlation values of the one of the one or more features and a favorable result of a judgement by a trained artificial intelligence model;
calculate a second value of correlation of the plurality of correlation values of the one of the one or more features and a unfavorable result of a judgement by the trained artificial intelligence model;
calculate an absolute value of a difference between the calculated first value and the calculated second value;
calculate a total sum of the calculated absolute value of the plurality of correlation values of the one of the one or more features; and
identify and monitor a bias in the one or more features using the calculated first value, the second value, the absolute value, and the total sum.

### Claim 14 (depends on 15)

The apparatus as set forth in claim 15 wherein the one or more processors are further configured to be capable of executing the programmed instructions stored in the memory to index feedback data received from a client device.

### Claim 15 (depends on 15)

The apparatus as set forth in claim 15 wherein the one or more processors are further configured to be capable of executing the programmed instructions stored in the memory to obtain training data from a training data server.

### Claim 16 (depends on 15)

The apparatus as set forth in claim 15 wherein the one or more processors are further configured to be capable of executing the programmed instructions stored in the memory to calculate a second total sum of the calculated absolute value of the plurality of correlation values of the one of the one or more features for the feedback data.

### Claim 17 (depends on 16)

The apparatus as set forth in claim 16 wherein the one or more processors are further configured to be capable of executing the programmed instructions stored in the memory to present a correlation change value determined by a difference between the calculated total sum of the calculated absolute value associated with the training data and the calculated total sum of the calculated absolute value associated with the feedback data.

### Claim 18 (depends on 17)

The apparatus as set forth in claim 17 wherein the correlation change value represents a candidate of a feature to be monitored.

### Claim 19 (depends on 13)

The apparatus as set forth in claim 13 wherein the one or more processors are further configured to be capable of executing the programmed instructions stored in the memory to provide the identified and monitored the one or more features with the bias to the client device.

### Claim 20 (depends on 19)

The apparatus as set forth in claim 19 wherein the one or more processors are further configured to be capable of executing the programmed instructions stored in the memory to provide a favorable or an unfavorable decision for the provided one or more features with the bias.

### Claim 21

A method for monitoring and detecting bias in an application, the method comprising:
identifying, by a bias monitoring computing system, a feature from one or more features of a software application;
determining, by the bias monitoring computing system, if there is a bias in the identified feature from the one or more features of the software application;
determining, by the bias monitoring computing system, a favorable or an unfavorable decision when there is the bias determined in the identified feature; and
providing, by the bias monitoring computing system, the identified feature with the determined bias and the determined favorable or an unfavorable decision.

### Claim 22 (depends on 21)

The method as set forth in claim 21 further comprising
indexing, by a bias monitoring computing system, training data; and
obtaining, by the bias monitoring computing system, a plurality of correlation values of one or more features in the indexed training data with a target variable.

### Claim 23 (depends on 22)

The method as set forth in claim 22 wherein, for each of the one or more features:
calculating, by the bias monitoring computing system, a first value of correlation of the plurality of correlation values of the one of the one or more features and a favorable result of a judgement by a trained artificial intelligence model; and
calculating, by the bias monitoring computing system, a second value of correlation of the plurality of correlation values of the one of the one or more features and a unfavorable result of a judgement by the trained artificial intelligence model.

### Claim 24 (depends on 23)

The method as set forth in claim 23 further comprising
calculating, by the bias monitoring computing system, an absolute value of a difference between the calculated first value and the calculated second value; and
calculating, by the bias monitoring computing system, a total sum of the calculated absolute value of the plurality of correlation values of the one of the one or more features.

### Claim 25 (depends on 24)

The method as set forth in claim 24 further comprising, identifying and monitoring, by the bias monitoring computing system, a bias in the one or more features using the calculated first value, the second value, the absolute value, and the total sum.
