# US 12,182,697 B2 · System and method for training artificial neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,182,697 B2
- Document ID: US12182697B2
- Status: Granted
- Kind: B2
- Application number: 16/223,055
- Filed: 2018-12-17
- Published: 2020-06-18
- Granted: 2024-12-31
- Earliest claimed: 2018-12-17
- Assignees: Integrated Silicon Solution, (Cayman) Inc.
- Applicants: Integrated Silicon Solution, (Cayman) Inc.
- Inventors: Michail Tzoufras; Marcin Gajek
- Web page: https://patentgrep.com/patent/US12182697B2

## CPC

- G06N 3/08: Learning methods
- G11C 11/54: using elements simulating biological cells, e.g. neuron
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0499: Feedforward networks
- G06N 3/063: using electronic means
- G06N 3/09: Supervised learning
- G06N 3/10: Interfaces, programming languages or software development kits, e.g. for simulating neural networks
- G11C 11/005: {comprising combined but independently operative RAM-ROM, RAM-PROM, RAM-EPROM cells}
- G11C 11/1673: {Reading or sensing circuits or methods}
- G11C 11/1675: {Writing or programming circuits or methods}
- G11C 11/16: using elements in which the storage effect is based on magnetic spin effect
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G11C: STATIC STORES (semiconductor memory devices H10B)

## Abstract

A computing device includes one or more processors, a first random access memory (RAM) comprising magnetic random access memory (MRAM), a second random access memory of a type distinct from MRAM, and a non-transitory computer-readable storage medium storing instructions for execution by the one or more processors. The computing device receives first data on which to train an artificial neural network (ANN) and trains the ANN by, using the first RAM comprising the MRAM, performing a first set of training iterations to train the ANN using the first data, and, after performing the first set of training iterations, using the second RAM of the type distinct from MRAM, performing a second set of training iterations to train the ANN using the first data. The computing device stores values for the trained ANN. The trained ANN is configured to classify second data based on the stored values.

## Claims

### Claim 1

A method, comprising:
performing, at a computing device that includes one or more processors, a first random access memory (RAM) comprising magnetic random access memory (MRAM), a second random access memory of a type distinct from MRAM, and a non-transitory computer-readable storage medium storing instructions for execution by the one or more processors, a set of operations that includes:
receiving first data on which to train an artificial neural network (ANN);
training the ANN by:
using the first RAM comprising the MRAM and operating the first RAM as an error-prone memory by operating the MRAM using a given read pulse duration or a given write pulse duration and without performing error correction to realize a first bit-error rate as a function of the read pulse duration or the write pulse duration, the first bit-error rate being greater than a bit-error rate of the second RAM, performing a first set of training iterations to train the ANN using only the first data and storing a set of weights and a set of biases of the ANN in the first RAM; and
after performing the first set of training iterations, using the second RAM of the type distinct from MRAM, performing a second set of training iterations to train the ANN using only the same first data and storing a set of weights and/or biases and a set of activations of the ANN in the second RAM; and
storing values for the trained ANN, wherein the trained ANN is configured to classify untrained, unclassified second data based on the stored values, the untrained, unclassified second data being distinct from the first data.

### Claim 2 (depends on 1)

The method of claim 1, wherein:
each of the first set of training iterations includes:
reading values for the set of weights of the ANN from the first RAM comprising the MRAM;
using the one or more processors, performing a set of arithmetic operations to update the values for the set of weights of the ANN; and
writing the updated set of weights of the ANN to the first RAM comprising the MRAM.

### Claim 3 (depends on 2)

The method of claim 2, wherein:
each of the first set of training iterations includes:
reading values for the set of biases of the ANN from the first RAM comprising the MRAM;
using the one or more processors, performing a set of arithmetic operations to update the values for the set of biases of the ANN; and
writing the updated set of biases of the ANN to the first RAM comprising the MRAM.

### Claim 4 (depends on 3)

The method of claim 3, wherein:
each of the second set of training iterations includes:
reading values for the set of weights and/or biases of the ANN from the second RAM of the type distinct from MRAM;
using the one or more processors, performing the set of arithmetic operations to update the values for the set of weights and/or biases of the ANN; and
writing the updated set of weights and/or biases of the ANN to the second RAM of the type distinct from the MRAM.

### Claim 5 (depends on 3)

The method of claim 3, wherein:
each of the second set of training iterations includes:
reading values for the set of activations of the ANN from the second RAM of the type distinct from MRAM;
using the one or more processors, performing the set of arithmetic operations to update the values for the set of activations of the ANN; and
writing the updated set of activations of the ANN to the second RAM of the type distinct from the MRAM.

### Claim 6 (depends on 1)

The method of claim 1, wherein the first RAM comprising the MRAM is on the same chip as the one or more processors.

### Claim 7 (depends on 1)

The method of claim 1, wherein the first RAM is operated, during the first set of training iterations, as the error-prone memory and the second RAM is operated, during the second set of training iterations, as an error-free memory.

### Claim 8 (depends on 7)

The method of claim 7, wherein:
the stored values of the trained ANN comprise stored weights;
the method further comprises:
during the first set of training iterations, performing error detection that includes detecting an error in a respective weight stored in the first RAM;
replacing a value stored in the respective weight with a zero value prior to using the respective weight.

### Claim 9 (depends on 7)

The method of claim 7, wherein the first RAM has a bit error-rate below a threshold for convergence of the first set of training iterations.

### Claim 10 (depends on 9)

The method of claim 9, wherein the threshold for convergence is greater than: 10−3, 10−5, or 10−7.

### Claim 11 (depends on 9)

The method of claim 9, wherein the bit-error rate is greater than: 10−4, 10−6, or 10−8.

### Claim 12 (depends on 1)

The method of claim 1, wherein the second RAM comprises static RAM (SRAM).

### Claim 13 (depends on 1)

The method of claim 1, wherein the first set of training iterations includes more than 20%, 40%, 60%, 80%, or 95% of a total number of training iterations used for training the ANN.

### Claim 14 (depends on 1)

The method of claim 1, further comprising, after training the ANN:
receiving the untrained, unclassified second data; and
assigning scores to the untrained, unclassified second data using the stored values of the trained ANN.

### Claim 15

An electronic system, comprising:
one or more processors;
a first random access memory (RAM) comprising magnetic random access memory (MRAM);
a second random access memory of a type distinct from MRAM;
a non-transitory computer-readable storage medium storing instructions executable by the one or more processors, including instructions for:
receiving first data on which to train an artificial neural network (ANN);
training the ANN by:
using the first RAM comprising the MRAM and operating the first RAM as an error-prone memory by operating the MRAM using a given read pulse duration or a given write pulse duration and without performing error correction to realize a first bit-error rate as a function of the read pulse duration or the write pulse duration, the first bit-error rate being greater than a bit-error rate of the second RAM, performing a first set of training iterations to train the ANN using only the first data and storing a set of weights and a set of biases of the ANN in the first RAM; and
after performing the first set of training iterations, using the second RAM of the type distinct from MRAM, performing a second set of training iterations to train the ANN using only the same first data and storing a set of weights and/or biases and a set of activations of the ANN in the second RAM; and
storing values for the trained ANN, wherein the trained ANN is configured to classify untrained, unclassified second data based on the stored values, the untrained, unclassified second data being distinct from the first data.

### Claim 16 (depends on 15)

The electronic system of claim 15, wherein:
each of the first set of training iterations includes:
reading values for the set of weights of the ANN from the first RAM comprising the MRAM;
using the one or more processors, performing a set of arithmetic operations to update the values for the set of weights of the ANN; and
writing the updated set of weights of the ANN to the first RAM comprising the MRAM.

### Claim 17 (depends on 16)

The electronic system of claim 16, wherein:
each of the first set of training iterations includes:
reading values for the set of biases of the ANN from the first RAM comprising the MRAM;
using the one or more processors, performing a set of arithmetic operations to update the values for the set of biases of the ANN; and
writing the updated set of biases of the ANN to the first RAM comprising the MRAM.

### Claim 18 (depends on 17)

The electronic system of claim 17, wherein:
each of the second set of training iterations includes:
reading values for the set of weights and/or biases of the ANN from the second RAM of the type distinct from MRAM;
using the one or more processors, performing the set of arithmetic operations to update the values for the set of weights and/or biases of the ANN; and
writing the updated set of weights and/or biases of the ANN to the second RAM of the type distinct from the MRAM.

### Claim 19 (depends on 17)

The electronic system of claim 17, wherein:
each of the second set of training iterations includes:
reading values for the set of activations of the ANN from the second RAM of the type distinct from MRAM;
using the one or more processors, performing the set of arithmetic operations to update the values for the set of activations of the ANN; and
writing the updated set of activations of the ANN to the second RAM of the type distinct from the MRAM.

### Claim 20 (depends on 15)

The electronic system of claim 15, further comprising, after training the ANN:
receiving the untrained, unclassified second data; and
assigning scores to the untrained, unclassified second data using the stored values of the trained ANN.