# US 2024/0004645 A1 · Intermediate Representation Controller Circuit for Selecting Hardware Compute Units to Process Microcode According to Identified Intermediate Representation Primitives

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0004645 A1
- Document ID: US20240004645A1
- Status: Published application
- Kind: A1
- Application number: 17/854,434
- Filed: 2022-06-30
- Published: 2024-01-04
- Earliest claimed: 2022-06-30
- Assignees: Advanced Micro Devices, Inc.
- Applicants: Advanced Micro Devices, Inc.
- Inventors: Sergey Blagodurov; Ganesh Suryanarayanc Dasika
- Web page: https://patentgrep.com/patent/US20240004645A1

## CPC

- G06F 9/22: Microcontrol or microprogram arrangements
- G06N 3/02: Neural networks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

An intermediate representation (IR) controller is described that, for a given intermediate representation (IR) primitive, selects a hardware compute unit of a plurality of hardware compute units. In a non-limiting example, the IR controller receives an input that specifies an IR primitive, a device mask indicating a type of hardware circuitry to be used to process the primitive, and a goal vector specifying a goal in the processing of the primitive. The IR controller also collects data describing power consumption by respective hardware compute units and completion times for processing respective IR primitives. This data is maintained as implementation profiles that describe operation of respective hardware compute units in processing respective IR primitives, e.g., as histograms. The implementation profiles are then leveraged by the IR controller to select hardware compute units for execution of subsequent IR primitives.

## Claims

### Claim 1

A method comprising:
receiving, by a controller circuit, from a neural network, an input identifying a goal vector and a specific intermediate representation (IR) primitive of a plurality of IR primitives, the goal vector defining a goal in how the specific IR primitive is to be processed;
identifying, by the controller circuit, at least one implementation profile from a plurality of implementation profiles based on the input, the plurality of implementation profiles describing operation of a plurality of microcode implementations in processing respective ones of the plurality of IR primitives;
selecting, by the controller circuit, a microcode implementation from the plurality of microcode implementations based on the at least one implementation profile and the goal; and
invoking, by the controller circuit, processing of microcode corresponding to the specific IR primitive by the selected microcode implementation.

### Claim 2 (depends on 1)

The method of claim 1, wherein the selecting of the microcode implementation from the plurality of microcode implementations is based at least in part on the goal in how the specific IR primitive is to be processed.

### Claim 3 (depends on 2)

The method of claim 2, wherein the goal in how the specific IR primitive is to be processed specifies whether performance or power efficiency is to be given a relatively higher priority when implementing the specific IR primitive.

### Claim 4 (depends on 1)

The method of claim 1, wherein the input further identifies a device mask specifying a type of hardware circuitry to be used to process the specific IR primitive and the identifying or the selecting is based at least in part on the type.

### Claim 5

(canceled)

### Claim 6 (depends on 1)

The method of claim 1, further comprising detecting, by the controller circuit, operating conditions of hardware compute units corresponding to the plurality of microcode implementations and wherein the selecting is based at least in part on the detected operating conditions.

### Claim 7 (depends on 6)

The method of claim 6, wherein the hardware compute units are implemented by a central processing unit, parallel processing unit, floating point grid array, or tensor processing unit.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
receiving, by the controller circuit, data describing operation of the selected microcode implementation in processing the microcode; and
updating, by the controller circuit, the at least one implementation profile based on the data.

### Claim 9 (depends on 1)

The method of claim 1, further comprising updating, by the controller circuit, one or more of the plurality of implementation profiles offline.

### Claim 10 (depends on 1)

The method of claim 1, wherein the plurality of implementation profiles are maintained as part of writeable microcode in a writeable control store.

### Claim 11

An intermediate representation (IR) controller circuit comprising:
an input module circuit configured to receive an input from a neural network, the input identifying a goal vector and an IR primitive, and the goal vector defining a goal in how the IR primitive is to be processed;
a profiler manager module circuit configured to collect data in a writeable control store as a plurality of implementation profiles, the plurality of implementation profiles describing operation of a plurality of hardware compute units in processing, respectively, a plurality of microcode implementations; and
an actuator module circuit configured to select, based on the goal, a specific hardware compute unit of the plurality of hardware compute units to process microcode corresponding to the IR primitive.

### Claim 12

(canceled)

### Claim 13 (depends on 11)

The IR controller circuit of claim 11, wherein the plurality of implementation profiles describe the operation using histograms.

### Claim 14 (depends on 13)

The IR controller circuit of claim 13, wherein the histograms describe power consumption or performance.

### Claim 15 (depends on 11)

The IR controller circuit of claim 11, wherein the actuator module circuit is configured to select the specific hardware compute unit based on operating conditions detected for the plurality of hardware compute units.

### Claim 16 (depends on 11)

The IR controller circuit of claim 11, wherein the goal in how the IR primitive is to be processed specifies whether performance or power efficiency is to be given a relatively higher priority when implementing the IR primitive.

### Claim 17 (depends on 11)

The IR controller circuit of claim 11, wherein the input further identifies a device mask specifying a type of hardware circuitry to be used to process the IR primitive and the actuator module circuit is configured to select the hardware compute unit from the plurality of hardware compute units based at least in part on the type.

### Claim 18

A method comprising:
generating a plurality of implementation profiles by an intermediate representation (IR) controller circuit based on data collected from and describing operation of a plurality of hardware compute units in processing microcode corresponding to an IR primitive;
forming an additional implementation profile by the IR controller circuit based on data collected from an additional hardware compute unit made available by communicatively coupling the additional hardware compute unit to the IR controller circuit;
receiving an input at the IR controller circuit to cause processing of the IR primitive;
determining by the IR controller circuit which hardware compute unit of the plurality of hardware compute units, including the additional hardware compute unit, is to be used to process the IR primitive based on the plurality of implementation profiles and the additional implementation profile; and
invoking processing of microcode corresponding to the IR primitive at the hardware compute unit by the IR controller circuit.

### Claim 19 (depends on 18)

The method of claim 18, wherein the forming, the receiving, the determining, and the invoking are performed in real time.

### Claim 20 (depends on 18)

The method of claim 18, wherein the generating is performed offline while at least a portion of the plurality of hardware compute units are idled.

### Claim 21 (depends on 18)

The method of claim 18, wherein receiving the input comprises receiving the input from machine learning software.

### Claim 22 (depends on 21)

The method of claim 21, wherein receiving the input comprises receiving the input from a neural network.
