# US 2022/0286399 A1 · Hardware queue scheduling for multi-core computing environments

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0286399 A1
- Document ID: US20220286399A1
- Status: Published application
- Kind: A1
- Application number: 17/637,416
- Filed: 2020-09-11
- Published: 2022-09-08
- Earliest claimed: 2019-09-11
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Niall McDonnell; Gage Eads; Mrittika Ganguli; Chetan Hiremath; John Mangan; Stephen Palermo; Bruce Richardson; Edwin Verplanke; Praveen Mosur; Bradley Chaddick; Abhishek Khade; Abhirupa Layek; Sarita Maini; Rahul Shah; Shrikant Shah; William Burroughs; David Sonnier
- Web page: https://patentgrep.com/patent/US20220286399A1

## CPC

- H04L 47/125: by balancing the load, e.g. traffic engineering
- H04L 47/62: characterised by scheduling criteria
- H04L 47/624: {Altering the ordering of packets in an individual queue}
- H04L 47/6255: {queue load conditions, e.g. longest queue first}
- H04L 47/6275: based on priority
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Methods, apparatus, systems, and articles of manufacture are disclosed for hardware queue scheduling for multi-core computing environments. An example apparatus includes a first core and a second core of a processor, and circuitry in a die of the processor, at least one of the first core or the second core included in the die, the at least one of the first core or the second core separate from the circuitry, the circuitry to enqueue an identifier to a queue implemented with the circuitry, the identifier associated with a data packet, assign the identifier in the queue to a first core of the processor, and in response to an execution of an operation on the data packet with the first core, provide the identifier to the second core to cause the second core to distribute the data packet.

## Claims

### Claim 1

An apparatus for hardware queue scheduling in a multi-core computing environment, the apparatus comprising:
a first core and a second core of a processor; and
circuitry in a die of the processor, at least one of the first core or the second core included in the die, the at least one of the first core or the second core separate from the circuitry, the circuitry to:
enqueue an identifier to a queue implemented with the circuitry, the identifier associated with a data packet;
assign the identifier in the queue to a first core of the processor; and
in response to an execution of an operation on the data packet with the first core, provide the identifier to the second core to cause the second core to distribute the data packet.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the queue is a first queue, and the circuitry is to:
determine a priority of the identifier based on the identifier;
dequeue the identifier from the first queue to first arbitration logic, the first arbitration logic associated with the priority;
provide the identifier from the first arbitration logic to second arbitration logic, the second arbitration logic associated with the first core; and
enqueue the identifier from the second arbitration logic to a second queue, the second queue associated with the first core, the first core to dequeue the identifier from the second queue.

### Claim 3 (depends on 1)

The apparatus of claim 1, wherein the identifier is a first identifier, the operation is a first operation, and the circuitry is to:
assign a second identifier in the queue to a third core of the processor in response to a throughput parameter not satisfying a throughput threshold, the throughput parameter based on telemetry data obtained from at least one of the first core or the second core, the second identifier associated with a second data packet; and
in response to the third core executing a second operation on the second data packet, provide the second identifier to the second core or a fourth core of the processor to cause the second core or the fourth core to distribute the second data packet.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein the queue is a first queue, and the circuitry is to:
receive, at a port of the circuitry, the identifier;
execute, with reordering logic of the circuitry, a reordering operation on the identifier;
identify, with arbitration logic of the circuitry, a second queue to enqueue the identifier; and
enqueue the identifier to the second queue, the first core to dequeue the identifier from the second queue.

### Claim 5 (depends on 4)

The apparatus of claim 4, wherein the identifier is a first identifier, the data packet is a first data packet, and the circuitry is to:
store the first identifier in an order buffer;
determine whether a second identifier is stored in the order buffer, the second identifier associated with a second data packet to be distributed after the first data packet;
in response to determining that the second identifier is stored in the order buffer, enqueue the first identifier in the queue; and
in response to enqueueing the first identifier in the queue, enqueue the second identifier in the queue.

### Claim 6 (depends on 1)

The apparatus of claim 1, wherein:
the first core is to:
provide a notification of a completion of the operation to the circuitry; and
store the data packet in memory; and
the second core is to retrieve the data packet from the memory in response to the circuitry providing the identifier to the second core.

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein the circuitry is to:
identify a data source of the identifier;
determine whether the data source has a producer credit, the identifier enqueued to the queue in response to determining that the data source has the producer credit;
deduct the producer credit from a number of producer credits associated with the data source, the number of producer credits stored in memory; and
in response to a distribution of the data packet, add the producer credit to the number of the producer credits.

### Claim at position 8

8-14. (canceled)

### Claim 15

An apparatus for hardware queue scheduling in a multi-core computing environment, the apparatus comprising:
means for enqueuing an identifier to a queue implemented with circuitry in a die of a processor, the identifier associated with a data packet;
means for assigning the identifier in the queue to a first core of the processor; and
means for allocating the identifier to a second core to cause the second core to distribute the data packet in response to an execution of an operation on the data packet with the first core, at least one of the first core or the second core are included in the die of the processor, the at least one of the first core or the second core separate from the circuitry.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein the queue is a first queue, and the circuitry is to:
determine a priority of the identifier based on the identifier;
dequeue the identifier from the first queue to first arbitration logic, the first arbitration logic associated with the priority;
provide the identifier from the first arbitration logic to second arbitration logic, the second arbitration logic associated with the first core; and
enqueue the identifier from the second arbitration logic to a second queue, the second queue associated with the first core, the first core to dequeue the identifier from the second queue.

### Claim 17 (depends on 15)

The apparatus of claim 15, wherein the identifier is a first identifier, the operation is a first operation, and:
the means for assigning is to assign a second identifier in the queue to a third core of the processor in response to a throughput parameter not satisfying a throughput threshold, the throughput parameter based on telemetry data obtained from at least one of the first core or the second core, the second identifier associated with a second data packet; and
the means for allocating is to allocate the second identifier to the second core or a fourth core of the processor to cause the second core or the fourth core to distribute the second data packet.

### Claim 18 (depends on 15)

The apparatus of claim 15, wherein the queue is a first queue, and further including:
means for receiving the identifier; and
means for executing a reordering operation on the identifier; and
the means for assigning is to identify a second queue to enqueue the identifier; and
the means for allocating is to enqueue the identifier to the second queue, the first core to dequeue the identifier from the second queue.

### Claim 19 (depends on 18)

The apparatus of claim 18, wherein the identifier is a first identifier, the data packet is a first data packet, and:
the means for executing is to:
store the first identifier in an order buffer; and
determine whether a second identifier is stored in the order buffer, the second identifier associated with a second data packet to be distributed after the first data packet; and
the means for enqueueing is to:
enqueue the first identifier in the queue in response to determining that the second identifier is stored in the order buffer; and
enqueue the second identifier in the queue in response to enqueuing the first identifier in the queue.

### Claim 20 (depends on 15)

The apparatus of claim 15, further including:
means for obtaining a notification of a completion of the operation from the first core, the first core to store the data packet in memory; and
the means for allocating is to allocate the identifier to the second core, the second core to retrieve the data packet from the memory in response to receiving the identifier.

### Claim 21 (depends on 15)

The apparatus of claim 15, further including means for managing a number of producer credits, the means for managing to:
identify a data source of the identifier;
determine whether the data source has a producer credit, the identifier enqueued to the queue in response to determining that the data source has the producer credit;
deduct the producer credit from a number of producer credits associated with the data source, the number of producer credits stored in memory; and
in response to a distribution of the data packet, add the producer credit to the number of the producer credits.

### Claim at position 16

22-28. (canceled)

### Claim 29

An apparatus for dynamic load balancing in a multi-core computing environment, the apparatus comprising:
a first core and a plurality of second cores of a processor; and
circuitry in a die of the processor, the circuitry separate from the first core and the second cores, the circuitry to:
enqueue identifiers in one or more queues in the circuitry, the identifiers associated with respective ones of data packets of a packet flow;
allocate one or more of the second cores to dequeue first ones of the identifiers in response to a throughput parameter of the first core not satisfying a throughput threshold to cause the one or more of the second cores to execute one or more operations on first ones of the data packets, the first ones of the data packets corresponding to the dequeued first ones of the identifiers; and
provide the first ones of the identifiers to one or more data consumers of the processor to distribute the first ones of the data packets.

### Claim 30 (depends on 29)

The apparatus of claim 29, wherein the circuitry is to:
allocate at least one of the first core or the one or more second cores of the processor to dequeue second ones of the identifiers in response to the throughput parameter not satisfying a throughput threshold to cause the at least one of the first core or the one or more second cores to execute one or more operations on second ones of the data packets, the second ones of the data packets corresponding to the dequeued second ones of the identifiers; and
provide the second ones of the identifiers to the one or more data consumers of the processor to distribute the second ones of the data packets.

### Claim 31 (depends on 29)

The apparatus of claim 29, wherein the data packets are first data packets, the packet flow is a first packet flow, the identifiers are first identifiers, the one or more operations are a first one or more operations, and the circuitry is to:
enqueue second identifiers in the one or more queues, the second identifiers associated with respective ones of second data packets of a second packet flow, the second packet flow different from the first packet flow;
allocate at least one of the first core or the one or more second cores to dequeue ones of the second identifiers to cause the at least one of the first core or the one or more second cores to execute second one or more operations on the second data packets, the second data packets corresponding to the dequeued ones of the second identifiers; and
provide the second identifiers to the one or more data consumers to distribute the second data packets.

### Claim 32

(canceled)

### Claim 33 (depends on 29)

The apparatus of claim 29, wherein the first core is a data producer, the data producer is to receive the data packets from a network interface in communication with a first network, and the one or more data consumers are to transmit the data packets to the network interface for distribution to a second network.

### Claim 34 (depends on 29)

The apparatus of claim 29, wherein the throughput parameter has a first value based on first telemetry data associated with the first core, the one or more second cores include a third core, and the circuitry is to:
allocate the third core to dequeue a first set of the first ones of the identifiers in response to the first value not satisfying the throughput threshold;
determine a second value of the throughput parameter in response to an allocation of the third core, the second value based on second telemetry data associated with at least one of the first core or the third core; and
dequeue a second set of the first ones of the identifiers to the first core and the third core in response to the second value satisfying the throughput threshold.

### Claim 35 (depends on 34)

The apparatus of claim 34, wherein the data packets are first data packets, the packet flow is a first packet flow, the identifiers are first identifiers, and the circuitry is to allocate one or more of second ones of the second cores to dequeue ones of second identifiers associated with second data packets of a second data flow different from the first packet flow, the second ones of the second cores not including the first core and the third core.

### Claim at position 24

36-44. (canceled)

### Claim 45

An apparatus for dynamic load balancing in a multi-core computing environment, the apparatus comprising:
a first core and a second core of a processor; and
circuitry in a die of the processor, the circuitry to:
enqueue an identifier in a queue in the circuitry, the identifier associated with a data packet of a packet flow;
dequeue the identifier to the first core to cause the first core to execute a first operation on the data packet;
enqueue the identifier in the queue in response to obtaining an indication of completion of the first operation;
dequeue the identifier to the first core or the second core to cause the first core or the second core to execute a second operation on the data packet different from the first operation; and
provide the identifier to a data consumer of the processor to distribute the data packet.

### Claim 46 (depends on 45)

The apparatus of claim 45, wherein the circuitry is separate from the first core and the second core, and the die includes the first core, the second core, and the circuitry.

### Claim 47 (depends on 45)

The apparatus of claim 45, wherein the die is a first die that includes the first core and the second core, and the processor includes a second die that includes the circuitry.

### Claim 48

(canceled)

### Claim 49 (depends on 45)

The apparatus of claim 45, wherein the queue is a first queue, the circuitry is to enqueue the identifier to a second queue from the first queue, the first core is to poll the second queue, and the first core is to dequeue the identifier from the second queue in response to the polling.

### Claim 50 (depends on 45)

The apparatus of claim 45, wherein the data packet is a first data packet, the packet flow is a first packet flow, the identifier is a first identifier, the queue is a first queue, and the circuitry is to:
enqueue a second identifier in a second queue, the second identifier associated with a second data packet of a second packet flow, the second packet flow different from the first packet flow;
allocate a third core of the processor to dequeue the second identifier to cause the third core to execute the first operation, the second operation, or a third operation on the second data packet, the second data packet corresponding to the dequeued second identifier; and
provide the second identifier to the data consumer to distribute the second data packet.

### Claim at position 31

51-58. (canceled)