# US 2020/0401344 A1 · Storage controller having data augmentation components for use with non-volatile memory die

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0401344 A1
- Document ID: US20200401344A1
- Status: Published application
- Kind: A1
- Application number: 16/718,148
- Filed: 2019-12-17
- Published: 2020-12-24
- Earliest claimed: 2019-06-20
- Assignees: Western Digital Technologies, Inc.
- Applicants: Western Digital Technologies, Inc.
- Inventors: Alexander Bazarsky; Ariel Navon
- Web page: https://patentgrep.com/patent/US20200401344A1

## CPC

- G06F 3/0659: {Command handling arrangements, e.g. command buffers, queues, command scheduling}
- G06T 1/60: Memory management
- G06F 3/0604: {Improving or facilitating administration, e.g. storage management}
- G06F 3/0688: {Non-volatile semiconductor memory arrays}
- G06F 11/076: {by exceeding a count or rate limit, e.g. word- or bit count limit}
- G06F 11/1044: {with specific ECC/EDC distribution}
- G06F 11/1048: {using arrangements adapted for a specific error detection or correction feature}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/063: using electronic means
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06T 3/20: Linear translation of whole images or parts thereof, e.g. panning
- G06T 3/608: by skew deformation, e.g. two-pass or three-pass rotation
- G06T 5/70: Denoising; Smoothing
- G06T 2207/10004: Still image; Photographic image
- G06T 2207/20172: Image enhancement details
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods and apparatus are disclosed for implementing data augmentation within a storage controller of a data storage device based on machine learning data read from a non-volatile memory (NVM) array of a memory die. Some particular aspects relate to configuring the storage controller to generate augmented versions of training images for use in training a Deep Learning Accelerator of an image recognition system by rotating, translating, skewing, cropping, etc., a set of initial training images obtained from a host device and stored in the NVM array. Other aspects relate to controlling components of the memory die to generate noise-augmented images by, for example, storing and then reading training images from worn regions of the NVM array to inject noise into the images. Data augmentation based on data read from multiple memory dies is also described, such as image data spread across multiple NVM arrays or multiple memory dies.

## Claims

### Claim 1

A data storage device, comprising:
a memory die with non-volatile memory (NVM) elements; and
a storage controller configured to
obtain machine learning data from the NVM elements of the memory die and
augment the machine learning data.

### Claim 2 (depends on 1)

The data storage device of claim 1, wherein the machine learning data comprise images and the storage controller is further configured to generate augmented images from one or more initial images obtained from the NVM elements.

### Claim 3 (depends on 2)

The data storage device of claim 2, wherein the storage controller is further configured to generate the augmented images by applying one or more of rotation, translation, skew, cropping, flipping, and noise addition to the one or more initial images obtained from the NVM elements to generate one or more altered images.

### Claim 4 (depends on 1)

The data storage device of claim 1,
wherein the storage controller comprises error correction components configured to correct errors in data read from the NVM elements of the memory die, and
wherein the storage controller is further configured to generate augmented data by controlling the error correction components to increase an amount of noise in data read from the NVM elements.

### Claim 5 (depends on 4)

The data storage device of claim 4, wherein the storage controller is further configured to generate the augmented data by reducing an amount of error correction performed by the error correction components compared to an amount of error correction that would otherwise be employed to read data not subject to data augmentation, and then reading stored data from the NVM elements of the memory die with the reduced error correction.

### Claim 6 (depends on 1)

The data storage device of claim 1,
wherein the memory die includes read components configured to apply read voltages to the NVM elements to read data from the NVM elements, and
wherein the storage controller is further configured to generate augmented data by controlling the memory die to modify the read voltages to retain a greater amount of read errors as compared to an amount of read errors that would otherwise occur when reading data not subject to data augmentation, and then controlling the memory die to apply the modified read voltages to the NVM elements while reading stored data from the NVM elements.

### Claim 7 (depends on 1)

The data storage device of claim 1, wherein the storage controller further comprises:
a worn NVM region identifier component configured to identify worn regions of the NVM elements of the memory die that are subject to storage errors; and
a worn region storage component configured to control the memory die to store un-augmented versions of data in the worn regions of the NVM elements of the memory die and then read the data from the worn regions of the NVM elements to obtain noise-augmented versions of the data.

### Claim 8 (depends on 1)

The data storage device of claim 1, wherein the NVM elements of the memory die comprise NAND storage elements.

### Claim 9 (depends on 1)

The data storage device of claim 1, further comprising a plurality of memory dies and wherein the storage controller is further configured to:
obtain machine learning data from the NVM elements of the plurality of memory dies in parallel; and
augment the machine learning data obtained from the plurality of memory dies.

### Claim 10 (depends on 9)

The data storage device of claim 9, wherein the storage controller is further configured to control the plurality of memory dies to adjust one or more of error correction within the memory dies, read voltages within the memory dies, and the storage of un-augmented versions of data in worn regions of the NVM elements of the memory dies.

### Claim 11

A method for use by a storage controller of a data storage device, the method comprising:
obtaining machine learning data from a non-volatile memory (NVM) array of a memory die; and
generating, at the storage controller, augmented machine learning data from the machine learning data.

### Claim 12 (depends on 11)

The method of claim 11, further comprising performing at least one machine learning operation with the augmented machine learning data using the storage controller.

### Claim 13 (depends on 12)

The method of claim 12, wherein performing the at least one machine learning operation using the storage controller further comprises training a deep learning accelerator (DLA) using the augmented machine learning data.

### Claim 14 (depends on 11)

The method of claim 11, wherein the machine learning data comprise labeled images and the augmented machine learning data comprise augmented labeled images.

### Claim 15 (depends on 14)

The method of claim 14, wherein generating the augmented machine learning data using the storage controller further comprises:
reading one or more of the labeled images from the NVM array; and
performing one or more of rotating, translating, skewing, cropping, flipping, and adding noise to the one or more labeled images to generate one or more altered images.

### Claim 16 (depends on 11)

The method of claim 11, wherein generating the augmented machine learning data using the storage controller further comprises:
reducing the use of error correction components that are configured to apply error correction to data read from the NVM array; and
reading machine learning data from the NVM array with reduced error correction.

### Claim 17 (depends on 11)

The method of claim 11, wherein generating the augmented machine learning data using the storage controller further comprises:
controlling the memory die to modify read voltages applied to the NVM elements as compared to read voltages that would otherwise be employed by read components to read images not subject to data augmentation; and
controlling the memory die to apply the modified read voltages to the NVM elements while reading machine learning data from the NVM elements.

### Claim 18 (depends on 11)

The method of claim 11, wherein generating the augmented machine learning data using the storage controller further comprises:
identifying portions of the NVM array subject to storage errors;
controlling the memory die to store the machine learning data in the portions of the NVM array subject to storage errors; and
reading the machine learning data from the portions of the NVM array subject to storage errors to obtain noise-augmented versions of the machine learning data.

### Claim 19 (depends on 11)

The method of claim 11, wherein the data storage device comprises a plurality of memory dies and wherein the method further comprises:
obtaining machine learning data from the NVM elements of the plurality of memory dies in parallel; and
augmenting the machine learning data obtained from the plurality of memory dies in parallel with augmented machine learning data.

### Claim 20

An apparatus for use with a data storage device, the apparatus comprising:
means within a storage controller of the data storage device for obtaining machine learning data from a non-volatile memory (NVM) array of a memory die of the data storage device; and
means within the storage controller of the data storage device for augmenting the machine learning data obtained from the NVM array with augmented machine learning data.