# US 10,872,290 B2 · Neural network processor with direct memory access and hardware acceleration circuits

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,872,290 B2
- Document ID: US10872290B2
- Status: Granted
- Kind: B2
- Application number: 15/711,457
- Filed: 2017-09-21
- Published: 2019-03-21
- Granted: 2020-12-22
- Earliest claimed: 2017-09-21
- Assignees: Raytheon Company
- Applicants: Raytheon Company
- Inventors: John R. Goulding; John E. Mixter; David R. Mucha
- Web page: https://patentgrep.com/patent/US10872290B2

## CPC

- G06N 3/063: using electronic means
- G06F 12/0824: {Distributed directories, e.g. linked lists of caches}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06F 12/1081: for peripheral access to main memory, e.g. direct memory access [DMA]
- G06F 13/28: using burst mode transfer, e.g. direct memory access {DMA}, cycle steal (G06F13/32 takes precedence)
- G11C 11/40603: {Arbitration, priority and concurrent access to memory cells for read/write or refresh operations}
- Y02D 10/00: Energy efficient computing, e.g. low power processors, power management or thermal management
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G11C: STATIC STORES (semiconductor memory devices H10B)
- Y02D: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE

## Abstract

A dynamically adaptive neural network processing system includes memory to store instructions representing a neural network in contiguous blocks, hardware acceleration (HA) circuitry to execute the neural network, direct memory access (DMA) circuitry to transfer the instructions from the contiguous blocks of the memory to the HA circuitry, and a central processing unit (CPU) to dynamically modify a linked list representing the neural network during execution of the neural network by the HA circuitry to perform machine learning, and to generate the instructions in the contiguous blocks of the memory based on the linked list.

## Claims

### Claim 1

A dynamically adaptive neural network processing system comprising:
memory to store a multi-level linked list and instructions determined based on the multi-level linked list in contiguous blocks of the memory, the instructions representing a neural network, the multi-level linked list including a first linked list representing a topology of the neural network and interconnections between neurons thereof, a second linked list indicating a memory location of the memory from which to begin transferring data to hardware acceleration (HA) circuitry in forward propagation of the neural network, and a third linked list including a starting memory address for each layer of the neural network in reverse order;
hardware acceleration (HA) circuitry, the HA circuitry to execute the instructions to implement the neural network;
direct memory access (DMA) circuitry to transfer the instructions from the contiguous blocks of the memory to the HA circuitry; and
a central processing unit (CPU) to dynamically modify the multi-level linked list during execution of the neural network by the HA circuitry and to generate the instructions in the contiguous blocks of the memory based on the multi-level linked list.

### Claim 2 (depends on 1)

The system of claim 1, wherein the DMA circuitry is further to transfer the instructions in the contiguous blocks of the memory corresponding to the modified multi-level linked list to the HA circuitry while the HA circuitry executes the neural network.

### Claim 3 (depends on 1)

The system of claim 1, wherein the DMA circuitry is further to transfer results of executing the neural network from the HA circuitry to the memory as contiguous data.

### Claim 4 (depends on 1)

The system of claim 1, wherein the CPU converts the multi-level linked list into a sequential array of neural instructions and/or data and the DMA circuitry is further to transfer the sequential array of neural instructions and/or data from the memory to the HA circuitry.

### Claim 5 (depends on 4)

The system of claim 4, wherein the HA circuitry includes a plurality of parallel processing elements to process in parallel a corresponding plurality of sequential blocks of the sequential array of neural instructions and/or data as organized by the CPU.

### Claim 6 (depends on 1)

The system of claim 1, wherein the HA circuitry includes static random access memory (SRAM) circuitry configured to store results of executing one layer of the neural network to be input to a next layer of the neural network executed by the HA circuitry.

### Claim 7

A method of executing a dynamically adaptive neural network, the method comprising:
creating and storing, by a central processing unit (CPU), a multi-level linked list and instructions determined based on the multi-level linked list in contiguous blocks of memory, the instructions representing a neural network, the multi-level linked list including a first linked list representing a topology of the neural network and interconnections between neurons thereof, a second linked list indicating a memory location of the memory from which to begin transferring data to hardware acceleration (HA) circuitry in forward propagation of the neural network, and a third linked list including a starting memory address for each layer of the neural network in reverse order;
transferring, by direct memory access (DMA) circuitry, the instructions from the contiguous blocks of the memory to hardware acceleration (HA) circuitry;
executing, by the HA circuitry, the neural network the instructions to implement the neural network;
dynamically modifying the multi-level linked list during execution of the neural network by the HA circuitry to perform machine learning; and
generating the instructions in the contiguous blocks of the memory, by the CPU, based on the multi-level linked list.

### Claim 8 (depends on 7)

The method of claim 7, further comprising transferring the instructions in the contiguous blocks of the memory corresponding to the modified multi-level linked list to the HA circuitry while the HA circuitry executes the neural network.

### Claim 9 (depends on 7)

The method of claim 7, further comprising transferring neural network execution results, by the DMA circuitry, from the HA circuitry to the memory as contiguous data.

### Claim 10 (depends on 7)

The method of claim 7, wherein the transferring comprises:
converting, by the CPU, the multi-level linked list into a sequential array of neural instructions and/or data; and
transferring, by the DMA circuitry, the sequential array of neural instructions and/or data from the contiguous blocks of the memory to the HA circuitry.

### Claim 11 (depends on 10)

The method of claim 10, wherein the executing the neural network comprises:
organizing, by the CPU, the sequential array of neural instructions and/or data into a plurality of sequential blocks; and
processing in parallel, by a plurality of parallel processing elements of the HA circuitry, the plurality of sequential blocks of the sequential array of neural instructions and/or data.

### Claim 12 (depends on 7)

The method of claim 7, wherein the executing comprises storing, by the HA circuitry, results of executing one layer of the neural network to be input to a next layer of the neural network in static random access memory (SRAM) circuitry of the HA circuitry.

### Claim 13 (depends on 7)

The method of claim 7, wherein the multi-level linked list includes read/write instructions for the DMA circuitry, and the transferring is according to the read/write instructions included in the multi-level linked list.

### Claim 14

At least one non-transitory machine-readable medium including instructions that, when executed by a machine, cause the machine to perform the following operations:
creating and storing, by a central processing unit (CPU), a multi-level linked list and instructions determined based on the multi-level linked list in contiguous blocks of memory, the instructions representing a neural network, the multi-level linked list including a first linked list representing a topology of the neural network and interconnections between neurons thereof, a second linked list indicating a memory location of the memory from which to begin transferring data to hardware acceleration (HA) circuitry in forward propagation of the neural network, and a third linked list including a starting memory address for each layer of the neural network in reverse order;
transferring, by direct memory access (DMA) circuitry, the instructions in the contiguous blocks of the memory to hardware acceleration (HA) circuitry;
executing, by the HA circuitry, the instructions to implement the neural network;
dynamically modifying the multi-level linked list, by the CPU, during execution of the neural network by the HA circuitry to perform machine learning; and
generating the instructions in the contiguous blocks of the memory, by the CPU, based on the multi-level linked list.

### Claim 15 (depends on 14)

The non-transitory machine-readable medium of claim 14, further including instructions that, when executed by the machine, cause the machine to perform the operation of transferring the instructions in the contiguous blocks of the memory corresponding to the modified multi-level linked list to the HA circuitry while the HA circuitry executes the neural network.

### Claim 16 (depends on 14)

The non-transitory machine-readable medium of claim 14, further including instructions that, when executed by the machine, cause the machine to perform the operation of transferring neural network execution results, by the DMA circuitry, from the HA circuitry to the memory as contiguous data.

### Claim 17 (depends on 14)

The non-transitory machine-readable medium of claim 14, wherein the operation of transferring comprises:
converting, by the CPU, the multi-level linked list into a sequential array of neural instructions and/or data; and
transferring, by the DMA circuitry, the sequential array of neural instructions and/or data from the contiguous blocks of the memory to the HA circuitry.

### Claim 18 (depends on 17)

The non-transitory machine-readable medium of claim 17, wherein the operation of executing the deep neural network comprises:
organizing, by the CPU, the sequential array of neural instructions and/or data into a plurality of sequential blocks; and
processing in parallel, by a plurality of parallel processing elements of the HA circuitry, the plurality of sequential blocks of the sequential array of neural instructions and/or data.

### Claim 19 (depends on 14)

The non-transitory machine-readable medium of claim 14, wherein the operation of executing the neural network comprises storing, by the HA circuitry, results of executing one layer of the neural network to be input to a next layer of the neural network in static random access memory (SRAM) circuitry of the HA circuitry.

### Claim 20 (depends on 14)

The non-transitory machine-readable medium of claim 14, wherein the multi-level linked list includes read/write instructions for the DMA circuitry, and the transferring is according to the read/write instructions included in the multi-level linked list.