# US 2022/0012575 A1 · Methods and apparatus for localized processing within multicore neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0012575 A1
- Document ID: US20220012575A1
- Status: Published application
- Kind: A1
- Application number: 17/367,512
- Filed: 2021-07-05
- Published: 2022-01-13
- Earliest claimed: 2020-07-09
- Assignees: Femtosense, Inc.
- Applicants: Femtosense, Inc.
- Inventors: Sam Brian Fok; Alexander Smith Neckar; Scott Henry Reid
- Web page: https://patentgrep.com/patent/US20220012575A1

## CPC

- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06F 9/3001: {Arithmetic instructions}
- G06F 9/30036: {Instructions to perform operations on packed data, e.g. vector, tile or matrix operations}
- G06F 9/3016: {Decoding the operand specifier, e.g. specifier format}
- G06F 9/3802: {Instruction prefetching}
- G06F 9/3818: {Decoding for concurrent execution}
- G06F 9/3836: {Instruction issuing, e.g. dynamic instruction scheduling or out of order instruction execution}
- G06F 9/3838: {Dependency mechanisms, e.g. register scoreboarding}
- G06F 9/3851: {from multiple instruction streams, e.g. multistreaming}
- G06F 9/3885: {using a plurality of independent parallel functional units}
- G06F 9/48: Program initiating; Program switching, e.g. by interrupt
- G06F 11/3024: {where the computing system component is a central processing unit [CPU]}
- G06F 11/3433: {for load management (allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029)}
- G06F 15/16: Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs {(coordinating program control therefor G06F9/52; in regulating and control system G05B)}
- G06F 15/7807: {System on chip, i.e. computer system on a single chip; System in package, i.e. computer system on one or more chips in a single package}
- G06F 16/901: Indexing; Data structures therefor; Storage structures (for retrieval from the web G06F16/951)
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/048: Activation functions
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/063: using electronic means
- G06N 3/10: Interfaces, programming languages or software development kits, e.g. for simulating neural networks
- H03M 7/3082: {Vector coding (for television signals, see H04N19/94)}
- H03M 7/6023: {Parallelization}
- G06N 3/08: Learning methods
- G06N 3/098: Distributed learning, e.g. federated learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H03M: CODING; DECODING; CODE CONVERSION IN GENERAL (using fluidic means F15C4/00; optical analogue/digital converters G02F7/00; coding, decoding or code conversion, specially adapted for particular applications, see the relevant subclasses, e.g. G01D, G01R, G06F, G06T, G09G, G10L, G11B, G11C, H04B, H04L, H04M, H04N; ciphering or deciphering for cryptography or other purposes involving the need for secrecy G09C)

## Abstract

Methods and apparatus for localized processing within multicore neural networks. Unlike existing solutions that rely on commodity software and hardware to perform “brute force” large scale neural network processing the various techniques described herein map and partition a neural network into the hardware limitations of a target platform. Specifically, the various implementations described herein synergistically leverage localization, sparsity, and distributed scheduling, to enable neural network processing within embedded hardware applications. As described herein, hardware-aware mapping/partitioning enhances neural network performance by e.g., avoiding pin-limited memory accesses, processing data in compressed formats/skipping unnecessary operations, and decoupling scheduling between cores.

## Claims

### Claim 1

A neural network processing apparatus, comprising:
a plurality of cores; and
one or more memories configured to store a first set of global parameters and a second set of local parameters;
wherein each core comprises logic configured to:
obtain a first input vector;
perform global neural network processing based on the first input vector and the first set of global parameters;
perform local neural network processing based on the second set of local parameters and a previous activation state vector of the core; and
generate a result vector for distribution to the plurality of cores.

### Claim 2 (depends on 1)

The neural network processing apparatus of claim 1, wherein the first set of global parameters comprises a portion of a global matrix associated with each core of the plurality of cores.

### Claim 3 (depends on 1)

The neural network processing apparatus of claim 1, further comprising logic configured to sparsify a dense vector to produce a resulting sparse vector.

### Claim 4 (depends on 1)

The neural network processing apparatus of claim 1, wherein:
each core further comprises logic configured to:
obtain one or more portions of a global activation vector from other cores of the plurality of cores;
create the global activation vector based on the one or more portions of the global activation vector from the other cores of the plurality of cores; and
perform the global neural network processing based on the global activation vector.

### Claim 5 (depends on 1)

The neural network processing apparatus of claim 1, wherein each core further comprises logic configured to broadcast the result vector to each other core of the plurality of cores.

### Claim 6 (depends on 1)

The neural network processing apparatus of claim 1, wherein each core further comprises logic configured to update the previous activation state vector of the core based on the result vector.

### Claim 7 (depends on 1)

The neural network processing apparatus of claim 1, wherein:
the one or more memories comprises a first memory and a second memory;
the first memory of the one or more memories is exclusively associated with a first core of the plurality of cores configured to store a first portion of the first set of global parameters and a second portion of the second set of local parameters associated with the first core; and
the second memory of the one or more memories is exclusively associated with a second core of the plurality of cores different from the first core of the plurality of cores and is configured to store a third portion of the first set of global parameters different from the first portion and a fourth portion of the second set of local parameters different from the second portion associated with the second core.

### Claim 8

A method of operating a core of a multicore neural network architecture comprising:
receiving a sparse activation vector;
retrieving a dense activation vector, a portion of a global weight matrix, and a local matrix;
calculating an updated dense activation vector based on the sparse activation vector, the dense activation vector, and the portion of the global weight matrix, and the local matrix;
calculating a portion of an updated sparse activation vector based on the updated dense activation vector; and
broadcasting the updated sparse activation vector to other cores of the multicore neural network architecture.

### Claim 9 (depends on 8)

The method of claim 8, wherein calculating the portion of the updated sparse activation vector includes applying a rectified linear activation function to the updated dense activation vector.

### Claim 10 (depends on 8)

The method of claim 8, wherein the dense activation vector, the portion of the global weight matrix, and the local matrix are retrieved from a memory associated with the core.

### Claim 11 (depends on 10)

The method of claim 10, wherein the memory is not associated with the other cores of the multicore neural network architecture.

### Claim 12 (depends on 8)

The method of claim 8, wherein calculating the portion of the updated sparse activation vector comprises:
concatenating a sparse input vector and a sparse global activation vector to form a concatenated sparse vector;
multiplying the portion of the global weight matrix with the concatenated sparse vector creating a first intermediate data structure;
multiplying the local matrix with the dense activation vector creating a second intermediate data structure;
performing a first sigmoid function on a first sum of a first section of the first intermediate data structure added to a second section of the second intermediate data structure creating a third intermediate data structure;
performing a second sigmoid function on a second sum of a third section of the first intermediate data structure added to a fourth section of the second intermediate data structure creating a fourth intermediate data structure;
performing a hyperbolic tangent function on a third sum of a fifth section of the first intermediate data structure and a first product of the third intermediate data structure and a sixth section of the second intermediate data structure creating a fifth intermediate data structure; and
calculating the updated dense activation vector based on a fourth sum of a second product of the fourth intermediate data structure and the sparse global activation vector and a third product of the fifth intermediate data structure and a difference of one and the fourth intermediate data structure.

### Claim 13 (depends on 8)

The method of claim 8, further comprising receiving a sparse input vector and the sparse activation vector, from a broadcast communication to a plurality of cores of the multicore neural network architecture.

### Claim 14 (depends on 8)

The method of claim 8, wherein broadcasting the updated sparse activation vector to the other cores of the multicore neural network architecture occurs asynchronous to broadcasts from the other cores.

### Claim 15

A non-transitory computer readable apparatus comprising a storage medium having one or more computer programs stored thereon, the one or more computer programs, when executed by a processing apparatus, being configured to:
obtain a first sparse vector;
perform global neural network processing based on the first sparse vector and a first set of global parameters;
perform local neural network processing based on a second set of local parameters and a dense vector that is specific to each core; and
sparsify the dense vector to generate a second sparse vector for broadcast to a plurality of cores.

### Claim 16 (depends on 15)

The non-transitory computer readable apparatus of claim 15, wherein the first sparse vector comprises an input vector and the first set of global parameters comprises a portion of a global matrix associated with a first core.

### Claim 17 (depends on 15)

The non-transitory computer readable apparatus of claim 15, wherein sparsifying the dense vector comprises skipping elements or adding null elements to the dense vector.

### Claim 18 (depends on 15)

The non-transitory computer readable apparatus of claim 15, wherein sparsifying the dense vector comprises applying a rectified linear activation function to the dense vector.

### Claim 19 (depends on 15)

The non-transitory computer readable apparatus of claim 15, wherein each core further comprises logic configured to broadcast the second sparse vector to each other core of the plurality of cores.

### Claim 20 (depends on 15)

The non-transitory computer readable apparatus of claim 15, wherein the one or more computer programs are further configured to broadcast the second sparse vector to the plurality of cores.
