# US 2021/0201195 A1 · Machine learning models based on altered data and systems and methods for training and using the same

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0201195 A1
- Document ID: US20210201195A1
- Status: Published application
- Kind: A1
- Application number: 16/854,107
- Filed: 2020-04-21
- Published: 2021-07-01
- Earliest claimed: 2019-12-30
- Assignees: Micron Technology, Inc.
- Applicants: Micron Technology, Inc.
- Inventors: Dmitry Vengertsev; Zahra Hosseinimakarem; Jonathan D. Harms
- Web page: https://patentgrep.com/patent/US20210201195A1

## CPC

- G06N 3/08: Learning methods
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/24: Classification techniques
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/09: Supervised learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06N 3/048: Activation functions
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Data may be abstracted and/or masked prior to being provided to a machine learning model for training. A machine learning model may provide a confidence level associated with a result. If the confidence level is too high, the machine learning model or an application including the machine learning model may refrain from providing the result as an output. In some examples, the machine learning model may provide a “second best” result that has an acceptable confidence level. In other examples, an error signal may be provided as the output. In accordance with examples of the present disclosure, data may be abstracted and/or masked prior to being provided to a machine learning model for training and confidence levels of results of the trained machine learning model may be used to determine when a result should be withheld.

## Claims

### Claim 1

An apparatus comprising:
a processor; and
a non-transitory computer readable medium configured to store an input received from another device, wherein the non-transitory computer readable medium is encoded with instructions that when executed by the processor cause the processor to:
generate from the input, using a machine learning model, a first value indicative of a result and second value indicative of a confidence level;
compare the second value to a threshold value, wherein the threshold value is indicative of the first value comprising a third value memorized from a training data set used to train the machine learning model; and
provide an output to the other device based on that comparing.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the output includes the result when the second value is equal to or less than the threshold value.

### Claim 3 (depends on 1)

The apparatus of claim 1, wherein the output includes an error signal when the second value is equal to or greater than the threshold value.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein the other device is a mobile device.

### Claim 5 (depends on 1)

The system of claim 1, wherein the machine learning model is trained using a modified training data set, wherein the modified training data set includes at least one of a masked value or an abstracted value.

### Claim 6 (depends on 5)

The system of claim 5, wherein at least some of the data from the modified training data set is received from the other device.

### Claim 7

A method comprising:
receiving a data input;
generating from the data input, using a machine learning model, a first value indicative of a result and a second value indicative of a confidence level;
comparing the second value to a threshold value, wherein the threshold value is indicative of the first value comprising a third value memorized from a training data set used to train the machine learning model; and
providing an output based on the comparing.

### Claim 8 (depends on 7)

The method of claim 7, wherein the output includes an error signal when the second value is equal to or above the threshold value.

### Claim 9 (depends on 7)

The method of claim 7, further comprising:
when the second value is equal to or above the threshold value:
analyzing the data input with the machine learning model to generate a fourth value indicative of a second result and a fifth value indicative of a second confidence level, wherein the fifth value is below the threshold value; and
providing the fourth value as the output.

### Claim 10 (depends on 7)

The method of claim 7, wherein the output includes the first value when the second value is equal to or below the threshold value.

### Claim 11 (depends on 7)

The method of claim 7, wherein the second value is generated based, at least in part, on a softmax algorithm.

### Claim 12 (depends on 7)

The method of claim 7, wherein the second value is generated based, at least in part, on a distribution of the first value.

### Claim 13 (depends on 7)

The method of claim 7, further comprising training the machine learning model using a modified training data set, wherein the modified training data set includes at least one of a masked value or an abstracted value.

### Claim 14

An apparatus comprising:
a processor; and
a non-transitory computer readable medium configured to store data, wherein the non-transitory computer readable medium is encoded with instructions that when executed by the processor, cause the processor to:
generate a first value indicative of a rank, wherein the first value indicates a number of times one or more values is included in the data;
generate a second value indicative of a classification, wherein the second value indicates one or more data types included in the data;
change a value of the one or more values included in the data based on the first value and the second value; and
generate a training data set comprising the data including the value changed.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein the instructions implement a machine learning model to generate the second value.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein the machine learning model is a neural network.

### Claim 17 (depends on 14)

The apparatus of claim 14, wherein the instructions implement a rules-based classification system to generate the second value.

### Claim 18 (depends on 14)

The apparatus of claim 14, wherein the non-transitory computer readable medium is further encoded with instructions to train a machine learning model with the training data set.

### Claim 19 (depends on 14)

The apparatus of claim 14, wherein the value of the one or more values is changed to an abstract value when the second value indicates the data type of the value is sensitive data and the first value indicates the number of times the value is included in the data is equal to or above a threshold value.

### Claim 20 (depends on 14)

The apparatus of claim 14, wherein the value of the one or more values is changed to a masked value when the second value indicates the data type of the value is sensitive data and the first value indicates the number of times the value is included in the data is equal to or below a threshold value.

### Claim 21

A method comprising:
receiving data;
generating a first value indicative of a rank, wherein the first value indicates a number of times one or more values is included in the data;
generating a second value indicative of a classification, wherein the second value indicates one or more data types included in the data;
changing a value of the one or more values included in the data based on the first value and the second value; and
generating a training data set comprising the data comprising the value changed.

### Claim 22 (depends on 21)

The method of claim 21, wherein changing the value comprises masking the value when the second value indicates the data type of the value is sensitive data and the first value indicates the number of times the value is included in the data is equal to or below a threshold value.

### Claim 23 (depends on 22)

The method of claim 22, wherein the sensitive data includes at least one of proper names, dates, addresses, passwords, birth dates, account numbers, or user names.

### Claim 24 (depends on 21)

The method of claim 21, wherein changing the value comprises abstracting the value when the second value indicates the data type of the value is sensitive data and the first value indicates the number of times the value is included in the data is equal to or above a threshold value.

### Claim 25 (depends on 21)

The method of claim 21, wherein the second value is generated by a rules-based classification.

### Claim 26 (depends on 21)

The method of claim 21, wherein the second value is generated by a machine learning model.

### Claim 27 (depends on 21)

The method of claim 21 further comprising training a machine learning model with the training data set.
