# US 2024/0004774 A1 · Neural network computing system and method of executing neural network model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0004774 A1
- Document ID: US20240004774A1
- Status: Published application
- Kind: A1
- Application number: 18/112,769
- Filed: 2023-02-22
- Published: 2024-01-04
- Earliest claimed: 2022-07-01
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Jungho KIM; Hyunjin KIM; Youngchan CHO; Hoon CHOI
- Web page: https://patentgrep.com/patent/US20240004774A1

## CPC

- G06F 11/3442: {for planning or managing the needed capacity}
- G06N 3/063: using electronic means
- G06F 1/324: by lowering clock frequency
- G06F 9/4887: {involving deadlines, e.g. rate based, periodic}
- G06F 9/5066: {Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs (mappping at compile time, see G06F8/451)}
- G06F 9/5094: {where the allocation takes into account power or heat criteria (power management in computers in general G06F1/3203; thermal management in computers in general G06F1/206)}
- G06N 3/10: Interfaces, programming languages or software development kits, e.g. for simulating neural networks
- G06F 2209/483: Multiproc
- Y02D 10/00: Energy efficient computing, e.g. low power processors, power management or thermal management
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- Y02D: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE

## Abstract

A neural network computing system includes a processor including heterogeneous computing devices configured to execute a neural network model; a memory configured to buffer input data and output data of the neural network model; a memory controller configured to control data input and data output of the memory; and a system bus configured to support communication between the processor and the memory controller. The processor determines a target execution time for each node included in the neural network model based on a target end-to-end execution time of the neural network model; controls operating frequencies of hardware devices including the heterogeneous computing devices, the memory controller, and the system bus based on a target computing device for execution of each node, an amount of work for each node, and the target execution time for each node, and executes the neural network model by operating at the operating frequencies.

## Claims

### Claim 1

A neural network computing system, comprising:
a processor comprising heterogeneous computing devices configured to execute a neural network model;
a memory configured to buffer input data and output data of the neural network model;
a memory controller configured to control data input and data output of the memory; and
a system bus configured to support communication between the processor and the memory controller,
wherein the processor is configured to:
determine a target execution time for each node among a plurality of nodes comprised in the neural network model based on a target end-to-end execution time of the neural network model;
control operating frequencies of hardware devices, including the heterogeneous computing devices, the memory controller, and the system bus, based on a target computing device for execution of each node, an amount of work for each node, and the target execution time for each node; and
execute the neural network model by operating at the controlled operating frequencies.

### Claim 2 (depends on 1)

The neural network computing system of claim 1, wherein the processor is further configured to:
adjust the amount of work for each node and the target execution time for each node based on an actual end-to-end execution time of the neural network model; and
adjust an operating frequency of each of the hardware devices based on the adjusted amount of work for each node and the adjusted target execution time for each node.

### Claim 3 (depends on 1)

The neural network computing system of claim 1, wherein the processor is further configured to:
determine an execution order of the plurality of nodes and whether to execute the plurality of nodes in parallel based on a connection relationship between the plurality of nodes and a plurality of edges included in the neural network model; and
determine the target execution time for each node by distributing the target end-to-end execution time of the neural network model to nodes, executed in sequence, in proportion to the amount of work for each node.

### Claim 4 (depends on 1)

The neural network computing system of claim 1, wherein the processor is further configured to determine the amount of work for each node based on a number of arithmetic instructions of each node, a number of memory instructions of each node, a complier option for each node, an amount of required memory bandwidth of each node, a size of input data for each node, and a size of output data for each node.

### Claim 5 (depends on 4)

The neural network computing system of claim 4, wherein the processor is further configured to determine the number of arithmetic instructions and the number of memory instructions of each node by analyzing source code included in each node.

### Claim 6 (depends on 4)

The neural network computing system of claim 4, wherein the processor is further configured to determine the amount of required memory bandwidth of each node based on the number of memory instructions, the size of the input data for each node, and the size of the output data for each node.

### Claim 7 (depends on 4)

The neural network computing system of claim 4, wherein the amount of work for each node comprises the number of arithmetic instructions of each node, the number of memory instructions of each node, an amount of work of the target computing device for each node determined based on a compiler option, an amount of memory access for each node determined based on the amount of required memory bandwidth of each node, the size of the input data for each node, and the size of the output data for each node.

### Claim 8 (depends on 7)

The neural network computing system of claim 7, wherein the processor is further configured to:
determine an amount of work of the heterogeneous computing devices required to execute the neural network model based on the amount of work of the target computing device for each node; and
determine an amount of work of the memory controller and an amount of work of the system bus required to execute the neural network model based on the amount of memory access for each node.

### Claim 9 (depends on 8)

The neural network computing system of claim 8, wherein the processor is further configured to:
determine a minimum operating frequency at which the target execution time for each node is complied with from among a plurality of operating frequencies for each of the hardware devices based on a performance of each of the hardware devices and an amount of work for each of the hardware devices; and
control the operating frequencies of the hardware devices based on the determined minimum operating frequency of each of the hardware devices.

### Claim 10 (depends on 1)

The neural network computing system of claim 1, wherein the heterogeneous computing devices comprise a central processing unit (CPU), a neural processing unit (NPU), a graphic processing unit (GPU), a digital signal processor (DSP), and an accelerator.

### Claim 11 (depends on 1)

The neural network computing system of claim 1, wherein the processor is further configured to receive the target end-to-end execution time of the neural network model as an input through an application programming interface (API).

### Claim 12

A neural network computing system comprising:
a processor comprising heterogeneous computing devices that are configured to execute a neural network model;
a memory configured to buffer input data and output data of the neural network model;
a memory controller configured to control data input and data output of the memory; and
a system bus configured to support communication between the processor and the memory controller,
wherein the processor comprises:
a feedforward control portion configured to determine operating frequencies of hardware devices, including the heterogeneous computing devices, the memory, and the memory controller, based on an input of a target end-to-end execution time of the neural network model and an input of static analysis data independent of a result of execution of the neural network model;
a neural network model execution portion configured to control the hardware devices according to the determined operating frequencies, execute the neural network model, and output an actual end-to-end execution time of the neural network model; and
a feedback control portion configured to adjust the operating frequencies of the hardware devices based on dynamic analysis data dependent on the result of the execution of the neural network model, including the actual end-to-end execution time.

### Claim 13 (depends on 12)

The neural network computing system of claim 12, wherein the feedback control portion is further configured to:
determine a degree of resource contention with an application simultaneously executed with the neural network model by analyzing a usage rate of each of the hardware devices while the neural network model is executed; and
adjust the operating frequency of each of the hardware devices based on the degree of resource contention.

### Claim 14 (depends on 13)

The neural network computing system of claim 13, wherein the feedback control portion is further configured to adjust the operating frequency of each of the hardware devices to increase based on a usage rate occupied by the neural network model among usage rates of the hardware devices decreases.

### Claim 15 (depends on 13)

The neural network computing system of claim 13,
wherein the neural network computing system further comprises a camera,
wherein the application is a camera application generating an image frame using the camera, and
wherein the neural network model comprises at least one of a model for sensing an object in the image frame, and a model for identifying what the object is.

### Claim 16 (depends on 12)

The neural network computing system of claim 12, wherein the feedback control portion is further configured to:
accumulate analysis data generated when the neural network model is repeatedly executed; and
generate the dynamic analysis data by processing the accumulated analysis data using one from among an average operation, an accumulative average operation, a simple moving average operation, a weighted moving average operation, and an exponential moving average operation of the accumulated analysis data.

### Claim 17 (depends on 12)

The neural network computing system of claim 12, wherein the feedforward control portion is further configured to:
estimate an amount of work of a target computing device for execution of each node, of a plurality of nodes includes in the neural network, and an amount of memory access for each node using the static analysis data;
determine a target execution time for each node based on the target end-to-end execution time of the neural network model;
estimate an amount of work of the heterogeneous computing devices based on the amount of work of the target computing device for each node;
estimate an amount of work of the memory controller and an amount of work of the system bus based on the amount of memory access for each node; and
determine an operating frequency of each of the hardware devices based on a performance of each of the hardware devices, the amount of work of the heterogeneous computing devices, the amount of work of the memory controller, and the amount of work of the system bus.

### Claim 18 (depends on 17)

The neural network computing system of claim 17, wherein the feedforward control portion is further configured to determine an operating frequency, as the operating frequency of each of the hardware devices, at which the target execution time for each node is complied with and power consumption is minimized from among operating frequencies of the hardware devices.

### Claim 19 (depends on 12)

The neural network computing system of claim 12, wherein the neural network model execution portion is further configured to release the operating frequencies of the hardware devices when execution of the neural network model is completed.

### Claim 20

A method of executing a neural network model, the method comprising:
estimating an amount of work for each node among a plurality of nodes included in the neural network model based on execution of the neural network model being triggered;
determining a target execution time for each node of the plurality of nodes based on a target end-to-end execution time of the neural network model;
controlling an operating frequency of each of a plurality of heterogeneous computing devices for executing the neural network model, based on a target computing device for each node, the amount of work for each node, and the target execution time for each node;
executing the neural network model using the plurality of heterogeneous computing devices operating according to the controlled operating frequency of each of the plurality of heterogeneous computing devices; and
adjusting the operating frequency of each of the plurality of heterogeneous computing devices for executing the neural network model based on an actual end-to-end execution time of the neural network model.
