# US 2021/0334338 A1 · Hardware Accelerated Discretized Neural Network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334338 A1
- Document ID: US20210334338A1
- Status: Published application
- Kind: A1
- Application number: 17/370,716
- Filed: 2021-07-08
- Published: 2021-10-28
- Earliest claimed: 2018-12-14
- Assignees: Western Digital Technologies, Inc.
- Applicants: Western Digital Technologies, Inc.
- Inventors: Wen Ma; Pi-Feng Chiu; Minghai Qin; Won Ho Choi; Martin Lueker-Boden
- Web page: https://patentgrep.com/patent/US20210334338A1

## CPC

- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G11C 11/54: using elements simulating biological cells, e.g. neuron
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/065: Analogue means
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G11C 13/0002: {using resistive RAM [RRAM] elements}
- G06G 7/163: using a variable impedance controlled by one of the input signals, variable amplification or transfer function {(G06G7/161, G06G7/162 take precedence)}
- G11C 13/003: {Cell access}
- G11C 2213/77: Array wherein the memory element being directly connected to the bit lines and word lines without any access device being used
- H03M 1/12: Analogue/digital converters ({H03M1/001 – } H03M1/10 take precedence)
- H03M 1/74: Simultaneous conversion
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G11C: STATIC STORES (semiconductor memory devices H10B)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06G: ANALOGUE COMPUTERS (analogue optical computing devices G06E3/00)
- H03M: CODING; DECODING; CODE CONVERSION IN GENERAL (using fluidic means F15C4/00; optical analogue/digital converters G02F7/00; coding, decoding or code conversion, specially adapted for particular applications, see the relevant subclasses, e.g. G01D, G01R, G06F, G06T, G09G, G10L, G11B, G11C, H04B, H04L, H04M, H04N; ciphering or deciphering for cryptography or other purposes involving the need for secrecy G09C)

## Abstract

An innovative low-bit-width device may include a first digital-to-analog converter (DAC), a second DAC, a plurality of non-volatile memory (NVM) weight arrays, one or more analog-to-digital converters (ADCs), and a neural circuit. The first DAC is configured to convert a digital input signal into an analog input signal. The second DAC is configured to convert a digital previous hidden state (PHS) signal into an analog PHS signal. NVM weight arrays are configured to compute vector matrix multiplication (VMM) arrays based on the analog input signal and the analog PHS signal. The NVM weight arrays are coupled to the first DAC and the second DAC. The one or more ADCs are coupled to the plurality of NVM weight arrays and are configured to convert the VMM arrays into digital VMM values. The neural circuit is configured to process the digital VMM values into a new hidden state.

## Claims

### Claim 1

A method comprising:
converting a digital input signal into an analog input signal;
computing, using a plurality of non-volatile memory (NVM) weight arrays, a plurality of vector matrix multiplication (VMM) arrays based on the analog input signal, wherein multiple parallel NVM cells in the plurality of NVM weight arrays represent one synaptic weight element;
converting the VMM arrays into digital VMM values; and
processing the digital VMM values through at least one activation function unit.

### Claim 2 (depends on 1)

The method of claim 1, wherein processing the digital VMM values through at least one activation function unit comprises:
processing the digital VMM values into a forget gate value, an input gate value, an output gate value, and a new candidate memory cell value;
calculating a hidden state based on the forget gate value, the input gate value, the output gate value, and the new candidate memory cell value; and
feeding the hidden state as an input for computing the plurality of VMM weight arrays during a next cycle.

### Claim 3 (depends on 2)

The method of claim 2, wherein calculating the hidden state further comprises:
calculating a new memory cell state; and
calculating the hidden state based on the new memory cell state.

### Claim 4 (depends on 1)

The method of claim 1, wherein the multiple parallel NVM cells include at least three parallel NVM cells per one synaptic weight element.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
averaging, before ADC quantization, redundant runs of the plurality of VMM arrays.

### Claim 6 (depends on 5)

The method of claim 5, wherein at least three redundant runs are used for averaging redundant runs of the plurality of VMM arrays.

### Claim 7 (depends on 5)

The method of claim 5, wherein at least five redundant runs are used for averaging redundant runs of the plurality of VMM arrays.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
averaging redundant runs of a plurality of activation function units to determine a plurality of activation function unit values; and
processing the plurality of activation function unit values through element-wise calculations to calculate a hidden state.

### Claim 9 (depends on 1)

The method of claim 1, wherein the NVM weight arrays comprise resistive cross-point arrays.

### Claim 10

A device comprising:
at least one digital-to-analog converter (DAC) configured to convert a digital input signal into an analog input signal;
a plurality of non-volatile memory (NVM) weight arrays configured to compute a plurality of vector matrix multiplication (VMM) arrays based on the analog input signal, wherein:
the plurality of NVM weight arrays is coupled to the at least one DAC; and
multiple parallel NVM cells in the plurality of NVM weight arrays represent one synaptic weight element;
at least one analog-to-digital converter (ADC) coupled to the plurality of NVM weight arrays, wherein the at least one ADC is configured to convert the VMM arrays into digital VMM values; and
a neural circuit configured to process the digital VMM values through at least one activation function unit.

### Claim 11 (depends on 10)

The device of claim 10, wherein the plurality of NVM weight arrays comprises a plurality of resistive cross-point arrays.

### Claim 12 (depends on 10)

The device of claim 10, wherein:
at least one array from the plurality of NVM weight arrays includes a plurality of junctions; and
each junction of the plurality of junctions includes multiple parallel NVM cells.

### Claim 13 (depends on 10)

The device of claim 10, wherein the multiple parallel NVM cells include at least three parallel NVM cells per one synaptic weight element.

### Claim 14 (depends on 10)

The device of claim 10, wherein:
the at least one ADC comprise a plurality of ADCs;
the neural circuit comprises a plurality of activation function units coupled to the plurality of ADCs; and
the plurality of activation function units is configured to receive and process the digital VMM values.

### Claim 15 (depends on 14)

The device of claim 14, wherein:
the neural circuit further comprises arithmetic circuitry coupled to the plurality of activation function units; and
the arithmetic circuitry is configured to generate a hidden state based on an output received from each activation function unit of the plurality of activation function units.

### Claim 16 (depends on 15)

The device of claim 15, further comprising:
at least one averaging component within the neural circuit and configured to average redundant runs of each activation function unit of the plurality of activation function units to determine a plurality of activation function unit values as output to the arithmetic circuitry.

### Claim 17 (depends on 10)

The device of claim 10, further comprising:
at least one analog integrate and average component situated between the plurality of NVM weight arrays and the at least one ADC and configured to average, before ADC quantization, redundant runs of the plurality of VMM arrays.

### Claim 18 (depends on 17)

The device of claim 17, wherein the at least one analog integrate and average component is configured to average at least three redundant runs for each VMM array of the plurality of VMM arrays.

### Claim 19 (depends on 17)

The device of claim 17, wherein the at least one analog integrate and average component is configured to average at least five redundant runs for each VMM array of the plurality of VMM arrays.

### Claim 20

A circuit, comprising:
means for converting a digital input signal into an analog input signal;
means for converting a digital previous hidden state (PHS) signal into an analog PHS signal;
means for computing a plurality of vector matrix multiplication (VMM) arrays based on the analog input signal and the analog PHS signal;
means for converting the VMM arrays into digital VMM values; and
means for processing the digital VMM values into a new hidden state.