# US 2023/0057792 A1 · Training data fidelity for machine learning applications through intelligent merger of curated auxiliary data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0057792 A1
- Document ID: US20230057792A1
- Status: Published application
- Kind: A1
- Application number: 17/408,384
- Filed: 2021-08-21
- Published: 2023-02-23
- Earliest claimed: 2021-08-21
- Assignees: AT&T Intellectual Property I, L.P.
- Applicants: AT&T Intellectual Property I, L.P.
- Inventors: Balachander Krishnamurthy; Subhabrata Majumdar
- Web page: https://patentgrep.com/patent/US20230057792A1

## CPC

- G06N 20/00: Machine learning
- G06F 16/215: Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

In one example, a method includes identifying a target performance metric of a machine learning algorithm, wherein the target performance metric is to be improved, obtaining a set of auxiliary data from a plurality of auxiliary data sources, wherein the plurality of auxiliary data sources is separate from a training data set used to train the machine learning algorithm, selecting a candidate attribute type from the set of auxiliary data, identifying a quality metric for the candidate attribute type, calculating a change in the target performance metric when data values associated with the candidate attribute type are included in the training data set, determining that a tradeoff between the target performance metric and the quality metric of the candidate attribute type is satisfied by inclusion of the data values in the training data set, and training the machine learning algorithm using the training data set augmented with the data value.

## Claims

### Claim 1

A method comprising:
identifying, by a processing system including at least one processor, a target performance metric of a machine learning algorithm, wherein the target performance metric comprises a performance metric that is to be improved;
obtaining, by the processing system, a set of auxiliary data from a plurality of auxiliary data sources, wherein the plurality of auxiliary data sources is separate from a source of a training data set that is used to train the machine learning algorithm;
selecting, by the processing system, a candidate attribute type from the set of auxiliary data;
identifying, by the processing system, a quality metric for the candidate attribute type;
calculating, by the processing system, a change in the target performance metric of the machine learning algorithm when data values associated with the candidate attribute type are included in the training data set;
determining, by the processing system, that a tradeoff between the target performance metric and the quality metric of the candidate attribute type is satisfied by inclusion of the data values associated with the candidate attribute type in the training data set; and
training, by the processing system, the machine learning algorithm using the training data set augmented with the data values associated with the candidate attribute type.

### Claim 2 (depends on 1)

The method of claim 1, wherein the target performance metric relates to a use case for the machine learning algorithm.

### Claim 3 (depends on 1)

The method of claim 1, wherein the target performance metric is identified in a signal provided by a human analyst.

### Claim 4 (depends on 1)

The method of claim 1, wherein the plurality of auxiliary data sources contains data that is not contained in the training data set.

### Claim 5 (depends on 1)

The method of claim 1, wherein the candidate attribute type is selected based on a use case for the machine learning algorithm.

### Claim 6 (depends on 5)

The method of claim 5, wherein the candidate attribute type is known to influence an improvement in the target performance metric for the use case.

### Claim 7 (depends on 1)

The method of claim 1, wherein the candidate attribute type is selected in response to a signal from a human analyst that instructs the processing system to select the candidate attribute type.

### Claim 8 (depends on 1)

The method of claim 1, wherein the quality metric indicates a degree of accuracy of the data values associated with the candidate attribute type.

### Claim 9 (depends on 8)

The method of claim 8, wherein the degree of accuracy is expressed as a confidence.

### Claim 10 (depends on 1)

The method of claim 1, wherein the quality metric is pre-computed prior to the processing system selecting the candidate attribute type.

### Claim 11 (depends on 1)

The method of claim 1, wherein the quality metric is computed by the processing system.

### Claim 12 (depends on 11)

The method of claim 11, wherein the quality metric is directly proportional to a number of the plurality of auxiliary data sources containing values that agree with the data values associated with the candidate attribute type.

### Claim 13 (depends on 11)

The method of claim 11, wherein the quality metric depends on how closely the data values associated with the candidate attribute type match crowdsourced values for the candidate attribute type.

### Claim 14 (depends on 11)

The method of claim 11, wherein the quality metric depends on how closely the data values associated with the candidate attribute type match values obtained from a focused survey delivered to a group of known subject matter experts.

### Claim 15 (depends on 1)

The method of claim 1, wherein the calculating comprises:
re-training, by the processing system, the machine learning algorithm with the data values associated with the candidate attribute type included in the training data set.

### Claim 16 (depends on 1)

The method of claim 1, wherein the candidate attribute type results in the change in the target performance metric being greater than other candidate attribute types which have been evaluated for inclusion in the training data set.

### Claim 17 (depends on 1)

The method of claim 1, wherein the determining is based on a consultation with a human analyst.

### Claim 18 (depends on 1)

The method of claim 1, wherein the tradeoff is satisfied when the change in the target performance metric is an improvement that at least meets a first threshold and when the quality metric of the candidate attribute type at least meets a second threshold.

### Claim 19

A non-transitory computer-readable medium storing instructions which, when executed by a processing system including at least one processor, cause the processing system to perform operations, the operations comprising:
identifying a target performance metric of a machine learning algorithm, wherein the target performance metric comprises a performance metric that is to be improved;
obtaining a set of auxiliary data from a plurality of auxiliary data sources, wherein the plurality of auxiliary data sources is separate from a source of a training data set that is used to train the machine learning algorithm;
selecting a candidate attribute type from the set of auxiliary data;
identifying a quality metric for the candidate attribute type;
calculating a change in the target performance metric of the machine learning algorithm when data values associated with the candidate attribute type are included in the training data set;
determining that a tradeoff between the target performance metric and the quality metric of the candidate attribute type is satisfied by inclusion of the data values associated with the candidate attribute type in the training data set; and
training the machine learning algorithm using the training data set augmented with the data values associated with the candidate attribute type.

### Claim 20

A device comprising:
a processing system including at least one processor; and
a non-transitory computer-readable medium storing instructions which, when executed by the processing system, cause the processing system to perform operations, the operations comprising:
identifying a target performance metric of a machine learning algorithm, wherein the target performance metric comprises a performance metric that is to be improved;
obtaining a set of auxiliary data from a plurality of auxiliary data sources, wherein the plurality of auxiliary data sources is separate from a source of a training data set that is used to train the machine learning algorithm;
selecting a candidate attribute type from the set of auxiliary data;
identifying a quality metric for the candidate attribute type;
calculating a change in the target performance metric of the machine learning algorithm when data values associated with the candidate attribute type are included in the training data set;
determining that a tradeoff between the target performance metric and the quality metric of the candidate attribute type is satisfied by inclusion of the data values associated with the candidate attribute type in the training data set; and
training the machine learning algorithm using the training data set augmented with the data values associated with the candidate attribute type.