# US 2021/0334631 A1 · Activation function processing method, activation function processing circuit, and neural network system including the same

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334631 A1
- Document ID: US20210334631A1
- Status: Published application
- Kind: A1
- Application number: 17/087,135
- Filed: 2020-11-02
- Published: 2021-10-28
- Earliest claimed: 2020-04-22
- Assignees: SK hynix Inc.
- Applicants: SK hynix Inc.
- Inventors: Yong Sang PARK; Joo Young KIM
- Web page: https://patentgrep.com/patent/US20210334631A1

## CPC

- G06N 3/048: Activation functions
- G06F 12/0292: {using tables or multilevel address translation means (G06F12/023 takes precedence; address translation in virtual memory systems G06F12/10)}
- G06N 3/063: using electronic means
- G06N 5/025: Extracting rules from data
- H03K 19/17728: Reconfigurable logic blocks, e.g. lookup tables
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H03K: PULSE TECHNIQUE (measuring pulse characteristics G01R; modulating sinusoidal oscillations with pulses H03C; transmission of digital information H04L; discriminator circuits detecting phase difference between two signals by counting or integrating cycles of oscillation H03D3/04; automatic control, starting, synchronisation or stabilisation of generators of electronic oscillations or pulses where the type of generator is irrelevant or unspecified H03L; coding, decoding or code conversion, in general H03M)

## Abstract

An activation function processing method includes processing a first activation function in a first mode by referring to a shared lookup table that includes a plurality of function values of the first activation function; and processing a second activation function in a second mode by referring to the shared lookup table, the second activation function being a different function than the first activation function.

## Claims

### Claim 1

A method for activation function processing, the method comprising:
processing a first activation function in a first mode by referring to a shared lookup table that includes a plurality of function values of the first activation function; and
processing a second activation function in a second mode by referring to the shared lookup table, the second activation function being a different function than the first activation function.

### Claim 2 (depends on 1)

The method of claim 1, wherein the processing of the second activation function in the second mode includes:
receiving an input value of the second activation function in the second mode;
converting the input value of the second activation function into an input value of the first activation function;
determining, by referring to the shared lookup table, a function value of the first activation function corresponding to the input value of the first activation function; and
converting the function value of the first activation function into a function value of the second activation function.

### Claim 3 (depends on 1)

The method of claim 1, wherein the processing of the first activation function in the first mode includes:
receiving an input value of the first activation function in the first mode; and
determining, by referring to the shared lookup table, a function value of the first activation function corresponding to the input value of the first activation function.

### Claim 4 (depends on 3)

The method of claim 3,
wherein the determining of the function value of the first activation function includes:
converting, according to an address conversion rule, the input value of the first activation function into a corresponding one of a plurality of addresses respectively indicating a plurality of entries of the shared lookup table, the plurality of function values being respectively included in the plurality of entries; and
determining, as the function value of the first activation function, a function value that is included in an entry indicated by the corresponding address within the shared lookup table, and
wherein the address conversion rule is set for each of all input values of the first activation function to be converted into a corresponding one of the plurality of addresses of the shared lookup table.

### Claim 5 (depends on 1)

The method of claim 1,
wherein the first activation function is a Sigmoid function;
wherein the second activation function is a Hyperbolic tangent function; and
wherein the shared lookup table includes a plurality of function values of the Sigmoid function.

### Claim 6 (depends on 1)

The method of claim 1,
wherein the first activation function is a Hyperbolic tangent function;
wherein the second activation function is a Sigmoid function; and
wherein the shared lookup table includes a plurality of function values of the Hyperbolic tangent function.

### Claim 7

A circuit, comprising:
a lookup table access unit configured to determine a function value of a first activation function corresponding to an input value of the first activation function by referring to a shared lookup table that includes a plurality of function values of the first activation function;
an input value conversion unit configured to convert an input value of a second activation function into the input value of the first activation function in a predetermined mode, the second activation function being a different function than the first activation function; and
a function value conversion unit configured to convert the function value of the first activation function into a function value of the second activation function in the predetermined mode.

### Claim 8 (depends on 7)

The circuit of claim 7, wherein, in the predetermined mode, the lookup table access unit receives the input value of the first activation function from the input value conversion unit and provide the function value conversion unit with the function value of the first activation function.

### Claim 9 (depends on 7)

The circuit of claim 7, wherein the lookup table access unit converts the input value of the first activation function into a corresponding one of a plurality of addresses respectively indicating a plurality of entries of the shared lookup table, the plurality of function values being respectively included in the plurality of entries, and determines, as the function value of the first activation function, a function value that is included in an entry indicated by the corresponding address within the shared lookup table.

### Claim 10 (depends on 9)

The circuit of claim 9,
wherein the lookup table access unit converts, according to an address conversion rule, the input value of the first activation function into the corresponding address,
wherein the address conversion rule is set for each of all input values of the first activation function to be converted into a corresponding one of the plurality of addresses of the shared lookup table, and
wherein the shared lookup table includes the plurality of function values selected from among all function values of the first activation function.

### Claim 11 (depends on 7)

The circuit of claim 7,
wherein the first activation function is a Sigmoid function;
wherein the second activation function is a Hyperbolic tangent function; and
wherein the shared lookup table includes a plurality of function values of the Sigmoid function.

### Claim 12 (depends on 11)

The circuit of claim 11, wherein the function value conversion unit converts a function value of the Sigmoid function into a function value of the Hyperbolic tangent function based on an equation:
tanh(x)=2σ(2x)−1,
wherein σ(2x) represents the function value of the Sigmoid function, tanh(x) represents the function value of the Hyperbolic tangent function, x is an input value of the Hyperbolic tangent function, and
wherein the circuit outputs the function value of the Hyperbolic tangent function as an output value in the predetermined mode.

### Claim 13 (depends on 7)

The circuit of claim 7,
wherein the first activation function is a Hyperbolic tangent function;
wherein the second activation function is a Sigmoid function; and
wherein the shared lookup table includes a plurality of function values of the Hyperbolic tangent function.

### Claim 14 (depends on 13)

The circuit of claim 13, wherein the function value conversion unit converts a function value of the Hyperbolic tangent function into a function value of the Sigmoid function based on an equation:
σ(x)=(tanh(x/2)+1)/2,
wherein σ(x) represents the function value of the Sigmoid function, tanh(x/2) represents the function value of the Hyperbolic tangent function, x is an input value of the Sigmoid function, and
wherein the circuit outputs the function value of the Sigmoid function as an output value in the predetermined mode.

### Claim 15

A neural network system, comprising:
an activation function processing unit configured to:
receive a first input value of a first activation function and determine a function value of the first activation function corresponding to the first input value by referring to a shared lookup table including a plurality of function values of the first activation function, and
receive a second input value of a second activation function and determine a function value of a second activation function corresponding to the second input value by referring to the shared lookup table, the first and second activation functions being different functions.

### Claim 16 (depends on 15)

The neural network system of claim 15, wherein the activation function processing unit includes a lookup table access unit configured to convert the first input value of the first activation function into a corresponding one of a plurality of addresses respectively indicating a plurality of entries of the shared lookup table, the plurality of function values being respectively included in the plurality of entries, and determine, as the function value of the first activation function corresponding to the first input value, a function value that is included in an entry indicated by the corresponding address within the shared lookup table.

### Claim 17 (depends on 16)

The neural network system of claim 16, wherein the activation function processing unit further includes an input value conversion unit configured to convert the second input value of the second activation function into the first input value of the first activation function in a predetermined mode.

### Claim 18 (depends on 17)

The neural network system of claim 17, wherein the activation function processing unit further includes a function value conversion unit configured to convert the function value of the first activation function into the function value of the second activation function in the predetermined mode.

### Claim 19 (depends on 15)

The neural network system of claim 15,
wherein the first activation function is a Sigmoid function;
wherein the second activation function is a Hyperbolic tangent function; and
wherein the shared lookup table includes a plurality of function values of the Sigmoid function.

### Claim 20 (depends on 15)

The neural network system of claim 15,
wherein the first activation function is a Hyperbolic tangent function;
wherein the second activation function is a Sigmoid function; and
wherein the shared lookup table includes a plurality of function values of the Hyperbolic tangent function.
