# US 2020/0401850 A1 · Non-volatile memory die with on-chip data augmentation components for use with machine learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0401850 A1
- Document ID: US20200401850A1
- Status: Published application
- Kind: A1
- Application number: 16/447,619
- Filed: 2019-06-20
- Published: 2020-12-24
- Earliest claimed: 2019-06-20
- Assignees: Western Digital Technologies, Inc.
- Applicants: Western Digital Technologies, Inc.
- Inventors: Alexander Bazarsky; Ariel Navon
- Web page: https://patentgrep.com/patent/US20200401850A1

## CPC

- G06N 3/08: Learning methods
- G06F 11/0727: {in a storage system, e.g. in a DASD or network based storage system (drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097)}
- G06F 11/1048: {using arrangements adapted for a specific error detection or correction feature}
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06T 3/02: Affine transformations (for image registration G06T3/147; for image mosaicing G06T3/4038)
- G06T 3/60: Rotation of whole images or parts thereof
- G06V 10/772: Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06V 10/955: {using specific electronic processors}
- G11C 16/26: Sensing or reading circuits; Data output circuits
- G11C 11/54: using elements simulating biological cells, e.g. neuron
- G11C 16/0483: {comprising cells having several storage transistors connected in series}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G11C: STATIC STORES (semiconductor memory devices H10B)

## Abstract

Methods and apparatus are disclosed for implementing machine learning data augmentation within the die of a non-volatile memory (NVM) apparatus using on-chip circuit components formed on or within the die. Some particular aspects relate to configuring under-the-array or next-to-the-array components of the die to generate augmented versions of images for use in training a Deep Learning Accelerator of an image recognition system by rotating, translating, skewing, cropping, etc., a set of initial training images obtained from a host device. Other aspects relate to configuring under-the-array or next-to-the-array components of the die to generate noise-augmented images by, for example, storing and then reading training images from worn regions of a NAND array to inject noise into the images.

## Claims

### Claim 1

An apparatus comprising:
a die with non-volatile memory (NVM) elements; and
a data augmentation controller formed in the die and configured to augment machine learning data stored within the NVM elements with augmented machine learning data.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the machine learning data comprise images and the data augmentation controller is further configured to generate augmented images from one or more initial images obtained from the NVM elements.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein the data augmentation controller is further configured to generate the augmented images by applying one or more of rotation, translation, skew, cropping, flipping, and noise addition to the one or more initial images obtained from the NVM elements to generate one or more altered images.

### Claim 4 (depends on 1)

The apparatus of claim 1,
wherein the apparatus further comprises error correction components configured to correct errors in data read from the NVM elements, and
wherein the data augmentation controller is further configured to generate augmented data by deactivating the error correction components and then reading stored data from the NVM elements without error correction.

### Claim 5 (depends on 1)

The apparatus of claim 1,
wherein the apparatus further comprises error correction components configured to correct errors in data read from the NVM elements, and
wherein the data augmentation controller is further configured to generate augmented data by reducing an amount of error correction performed by the error correction components compared to an amount of error correction that would otherwise be employed to read data not subject to data augmentation, and then reading stored data from the NVM elements with the reduced error correction.

### Claim 6 (depends on 1)

The apparatus of claim 1,
wherein the apparatus further comprises read components configured to apply read voltages to the NVM elements to read data from the NVM elements, and
wherein the data augmentation controller is further configured to generate augmented data by modifying the read voltages to retain a greater amount of read errors as compared to an amount of read errors that would otherwise occur when reading data not subject to data augmentation, and then applying modified read voltages to the NVM elements while reading stored data from the NVM elements.

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein the data augmentation controller further comprises:
a worn NVM region identifier component configured to identify worn regions of the NVM elements that are subject to storage errors; and
a worn region storage component configured to store un-augmented versions of data in the worn regions of the NVM elements and then read the data from the worn regions of the NVM elements to obtain noise-augmented versions of the data.

### Claim 8 (depends on 1)

The apparatus of claim 1, wherein the NVM elements comprise NAND flash storage elements.

### Claim 9 (depends on 1)

The apparatus of claim 1, wherein the data augmentation controller formed in the die is further configured as one or more of an under-the-array component and a next-to-the-array component.

### Claim 10 (depends on 1)

The apparatus of claim 1, further comprising machine learning components formed in the die and configured to perform machine learning using the augmented machine learning data.

### Claim 11

A method for use with a die having a non-volatile memory (NVM) array, the method comprising:
storing machine learning data within the NVM array of the die; and
generating augmented machine learning data using data augmentation circuitry formed in the die.

### Claim 12 (depends on 11)

The method of claim 11, further comprising performing at least one machine learning operation using the augmented machine learning data using a machine learning controller formed in the die.

### Claim 13 (depends on 12)

The method of claim 12, wherein performing the at least one machine learning operation further comprises training a deep learning accelerator (DLA) using the augmented machine learning data.

### Claim 14 (depends on 11)

The method of claim 11, wherein the machine learning data comprise labeled images and the augmented machine learning data comprise augmented labeled images.

### Claim 15 (depends on 14)

The method of claim 14, wherein generating the augmented machine learning data further comprises:
reading one or more of the labeled images from the NVM array; and
performing one or more of rotating, translating, skewing, cropping, flipping, and adding noise to the one or more labeled images to generate one or more altered images.

### Claim 16 (depends on 11)

The method of claim 11, wherein generating the augmented machine learning data further comprises:
reducing the use of error correction components that are configured to apply error correction to data read from the NVM array; and
reading machine learning data from the NVM array with reduced error correction.

### Claim 17 (depends on 11)

The method of claim 11, wherein generating the augmented machine learning data further comprises:
modifying read voltages applied to the NVM elements as compared to read voltages that would otherwise be employed by read components to read images not subject to data augmentation; and
applying the modified read voltages to the NVM elements while reading machine learning data from the NVM elements.

### Claim 18 (depends on 11)

The method of claim 11, wherein generating the augmented machine learning data further comprises:
identifying portions of the NVM array subject to storage errors;
reading machine learning data from a first portion of the NVM array;
storing the machine learning data in the portions of the NVM array subject to storage errors; and
reading the machine learning data from the portions of the NVM array subject to storage errors to obtain noise-augmented versions of the machine learning data.

### Claim 19

An apparatus for use with a die having a non-volatile memory (NVM) array, the apparatus comprising:
means formed in the die for storing machine learning data within the NVM array of the die; and
means formed in the die for generating at least one augmented version of the machine learning data.

### Claim 20 (depends on 19)

The apparatus of claim 19, wherein the means for generating at least one augmented version of the machine learning data comprises means for generating augmented images by applying one or more of rotation, translation, skew, cropping, flipping, and noise addition to one or more initial images to generate one or more altered images.