# US 12,112,112 B2 · Method for co-design of hardware and neural network architectures using coarse-to-fine search, two-phased block distillation and neural hardware predictor > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,112 B2 - Document ID: US12112112B2 - Status: Granted - Kind: B2 - Application number: 17/095,937 - Filed: 2020-11-12 - Published: 2022-05-12 - Granted: 2024-10-08 - Earliest claimed: 2020-11-12 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Niv Zehngut; Amir Ben-Dror; Evgeny Artyomov; Michael Dinerstein; Roy Jevnisek - Web page: https://patentgrep.com/patent/US12112112B2 ## CPC - G06F 30/39: Circuit design at the physical level (physical level design for reconfigurable circuits G06F30/347) - G06F 30/27: using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model - G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections - G06N 3/086: using evolutionary algorithms, e.g. genetic algorithms or genetic programming - G06N 3/09: Supervised learning - G06N 3/092: Reinforcement learning - G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06F 2117/04: Clock gating - G06F 2117/08: HW-SW co-design, e.g. HW-SW partitioning - G06F 2119/06: Power analysis or power optimisation - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Methods, systems, and apparatus for combined or separate implementation of coarse-to-fine neural architecture search (NAS), two-phase block NAS, variable hardware prediction, and differential hardware design are provided and described. A variable predictor is trained, as described herein. Then, a controller or policy may be used to iteratively modify a neural network architecture along dimensions formed by neural network architecture parameters. The modification is applied to blocks (e.g., subnetworks) within the neural network architecture. In each iteration, the remainder of the neural network architecture parameters are modified and learned with a differential NAS method. The training process is performed with two-phase block NAS and incorporates a variable hardware predictor to predict power, performance, and area (PPA) parameters. The hardware parameters may be learned as well using the variable hardware predictor. ## Claims ### Claim 1 A method comprising: identifying a plurality of dimensions of a neural architecture search into a first group of dimensions including a coarse dimension and a second group of dimensions including a fine dimension; selecting a value of the coarse dimension during a coarse search phase of a neural architecture search using a controller based search algorithm, wherein the controller based search algorithm for the coarse search phase includes iteratively altering the value of the coarse dimension using a policy function; identifying a search space corresponding to the fine dimension from the second group of dimensions for a fine search phase of the neural architecture search based at least in part on the value of the coarse dimension selected during the coarse search phase of the neural architecture search; selecting a value for the fine dimension during the fine search phase of the neural architecture search using a differential search algorithm, wherein the differential search algorithm is based on a loss function; and generating a neural network architecture based on the value of the coarse dimension and the value of the fine dimension. ### Claim 2 (depends on 1) The method of claim 1, wherein: the coarse dimension comprises a number of layers, a layer type, a layer connectivity, or any combination thereof. ### Claim 3 (depends on 1) The method of claim 1, wherein: the fine dimension comprises a layer size, a number of channels, a filter size, a cell size, a gate size, an activation function, or any combination thereof. ### Claim 4 (depends on 1) The method of claim 1, wherein: the controller based search algorithm comprises a reinforcement learning algorithm or an evolutionary search algorithm. ### Claim 5 (depends on 1) The method of claim 1, further comprising: configuring a block of a neural network based on the neural architecture search; computing a first loss function based on an intermediate output of the block; updating the neural architecture search for the block based on the first loss function; computing a second loss function using a final output of the neural network after updating neural architecture search based on the first loss function; and updating the neural architecture search for the neural network based on the second loss function. ### Claim 6 (depends on 1) The method of claim 1, further comprising: identifying neural architecture parameters including the value of the coarse dimension and the value of the fine dimension; identifying a set of hardware parameters; predicting power, performance, and area (PPA) parameters using a variable hardware predictor; iteratively updating the neural architecture parameters and the hardware parameters based on the predicted PPA parameters; and generating a hardware design based on the updated hardware parameters. ### Claim 7 (depends on 6) The method of claim 6, further comprising: selecting the hardware parameters using a neural network trained using a differential training algorithm. ### Claim 8 An apparatus comprising a processor and a memory storing instructions and in electronic communication with the processor, the processor being configured to execute the instructions to: identify a plurality of dimensions of a neural architecture search into a first group of dimensions including a coarse dimension and a second group of dimensions including a fine dimension; select a value of the coarse dimension during a coarse search phase of a neural architecture search using a controller based search algorithm, wherein the controller based search algorithm for the coarse search phase includes iteratively altering the value of the coarse dimension using a policy function; identify a search space corresponding to the fine dimension from the second group of dimensions for a fine search phase of the neural architecture search based at least in part on the value of the coarse dimension selected during the coarse search phase of the neural architecture search; select a value for the fine dimension during the fine search phase of the neural architecture search using a differential search algorithm, wherein the differential search algorithm is based on a loss function; and generate a neural network architecture based on the value of the coarse dimension and the value of the fine dimension. ### Claim 9 (depends on 8) The apparatus of claim 8, wherein: the coarse dimension comprises a number of layers, a layer type, a layer connectivity, or any combination thereof. ### Claim 10 (depends on 8) The apparatus of claim 8, wherein: the fine dimension comprises a layer size, a number of channels, a filter size, a cell size, a gate size, an activation function, or any combination thereof. ### Claim 11 (depends on 8) The apparatus of claim 8, wherein: the controller based search algorithm comprises a reinforcement learning algorithm or an evolutionary search algorithm. ### Claim 12 (depends on 8) The apparatus of claim 8, the processor being further configured to execute the instructions to: configure a block of a neural network based on the neural architecture search; compute a first loss function based on an intermediate output of the block; update the neural architecture search for the block based on the first loss function; compute a second loss function using a final output of the neural network after updating neural architecture search based on the first loss function; and update the neural architecture search for the neural network based on the second loss function. ### Claim 13 (depends on 8) The apparatus of claim 8, the processor being further configured to execute the instructions to: identify neural architecture parameters including the value of the coarse dimension and the value of the fine dimension; identify a set of hardware parameters; predict power, performance, and area (PPA) parameters using a variable hardware predictor; iteratively update the neural architecture parameters and the hardware parameters based on the predicted PPA parameters; and generate a hardware design based on the updated hardware parameters. ### Claim 14 (depends on 13) The apparatus of claim 13, the processor being further configured to execute the instructions to: select the hardware parameters using a neural network trained using a differential training algorithm. ### Claim 15 A non-transitory computer readable medium storing code, the code comprising instructions executable by a processor to: identifying a plurality of dimensions of a neural architecture search into a first group of dimensions including a coarse dimension and a second group of dimensions including a fine dimension; select a value of the coarse dimension during a coarse search phase of a neural architecture search using a controller based search algorithm, wherein the controller based search algorithm for the coarse search phase includes iteratively altering the value of the coarse dimension using a policy function; identify a search space corresponding to the fine dimension from the second group of dimensions for a fine search phase of the neural architecture search based at least in part on the value of the coarse dimension selected during the coarse search phase of the neural architecture search; select a value for the fine dimension during the fine search phase of the neural architecture search using a differential search algorithm, wherein the differential search algorithm is based on a loss function; and generate a neural network architecture based on the value of the coarse dimension and the value of the fine dimension. ### Claim 16 (depends on 15) The non-transitory computer readable medium of claim 15, wherein: the coarse dimension comprises a number of layers, a layer type, a layer connectivity, or any combination thereof. ### Claim 17 (depends on 15) The non-transitory computer readable medium of claim 15, wherein: the fine dimension comprises a layer size, a number of channels, a filter size, a cell size, a gate size, an activation function, or any combination thereof. ### Claim 18 (depends on 15) The non-transitory computer readable medium of claim 15, wherein: the controller based search algorithm comprises a reinforcement learning algorithm or an evolutionary search algorithm. ### Claim 19 (depends on 15) The non-transitory computer readable medium of claim 15, the code further comprising instructions executable by the processor to: configure a block of a neural network based on the neural architecture search; compute a first loss function based on an intermediate output of the block; update the neural architecture search for the block based on the first loss function; compute a second loss function using a final output of the neural network after updating neural architecture search based on the first loss function; and update the neural architecture search for the neural network based on the second loss function. ### Claim 20 (depends on 15) The non-transitory computer readable medium of claim 15, the code further comprising instructions executable by the processor to: identify neural architecture parameters including the value of the coarse dimension and the value of the fine dimension; identify a set of hardware parameters; predict power, performance, and area (PPA) parameters using a variable hardware predictor; iteratively update the neural architecture parameters and the hardware parameters based on the predicted PPA parameters; and generate a hardware design based on the updated hardware parameters.