# US 2021/0279832 A1 · Completion Signaling Techniques in Distributed Processor > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0279832 A1 - Document ID: US20210279832A1 - Status: Published application - Kind: A1 - Application number: 16/812,724 - Filed: 2020-03-09 - Published: 2021-09-09 - Earliest claimed: 2020-03-09 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Andrew M. Havlir; Ajay Simha Modugala - Web page: https://patentgrep.com/patent/US20210279832A1 ## CPC - G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining - G06F 9/4818: {Priority circuits therefor} - G06F 9/505: {considering the load} - G06T 15/005: {General purpose rendering architectures} - Y02D 10/00: Energy efficient computing, e.g. low power processors, power management or thermal management - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - Y02D: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE ## Abstract Techniques are disclosed relating to tracking compute workgroup completions in a distributed processor. In some embodiments, an apparatus includes a plurality of shader processors configured to perform operations for compute workgroups included in compute kernels, a master workload parser circuit, a plurality of distributed workload parser circuits, and a communications fabric connected to the plurality of distributed workload parser circuits and the master workload parser circuit. In some embodiments, a distributed workload parser circuit is configured to maintain, for each of a set of the shader processors, a data structure that specifies a count of workgroup completions for one or more kernels processed by the shader processor, determine, for the set of shader processors based on counts of workgroup completions for a first kernel, an aggregate count of completions to report for the first kernel, send the aggregate count to the master workload parser circuit over the communications fabric, and adjust the data structures to reflect counts included in the aggregate count. ## Claims ### Claim 1 An apparatus, comprising: a plurality of shader processors configured to perform operations for compute workgroups included in compute kernels processed by the apparatus; a master workload parser circuit; a plurality of distributed workload parser circuits; a communications fabric connected to the plurality of distributed workload parser circuits and the master workload parser circuit; where at least one of the distributed workload parser circuits is configured to: maintain, for each of a set of the shader processors, a data structure that specifies a count of workgroup completions for one or more kernels processed by the shader processor; determine, for the set of shader processors based on counts of workgroup completions for a first kernel, an aggregate count of completions to report for the first kernel; send the aggregate count to the master workload parser circuit over the communications fabric; and adjust the data structures to reflect counts included in the aggregate count. ### Claim 2 (depends on 1) The apparatus of claim 1, wherein the data structure is a first-in first-out (FIFO) structure that includes an entry for each kernel for which a shader processor has completions and a corresponding completion count for that kernel. ### Claim 3 (depends on 2) The apparatus of claim 2, wherein the distributed workload parser circuit is configured to determine the aggregate count based on data in the oldest entry in each FIFO. ### Claim 4 (depends on 3) The apparatus of claim 3, wherein the apparatus is configured to maintain priority information for one or more of the FIFOs and wherein the distributed workload parser circuit is configured to select to report completions for the first kernel based on the first kernel being represented by the oldest entry in a shader FIFO having a current highest priority. ### Claim 5 (depends on 1) The apparatus of claim 1, wherein the distributed workload parser circuits include respective completion aggregator circuitry configured to determine the aggregate count, wherein the completion aggregator circuitry includes: a chain of aggregator circuits including one or more aggregator circuits configured to: receive: a completion count from a corresponding data structure; an accumulated count from one or more prior aggregator circuits in the chain; and a kernel identifier of the kernel for which counts are being aggregated; and generate: an updated accumulated count based on the received completion count and the received accumulated count; and a signal indicating a number of completions accepted from the corresponding data structure. ### Claim 6 (depends on 5) The apparatus of claim 5, wherein the completion aggregator circuitry is configured to determine a current highest-priority shader processor; and wherein the completion aggregator circuitry further includes multiplexer circuitry configured to route information from the data structure of the highest-priority shader to a starting aggregator circuit in the chain of aggregator circuits and route the signals indicating the numbers of accepted completions to corresponding shader processors. ### Claim 7 (depends on 5) The apparatus of claim 5, wherein the apparatus is configured to adjust a count value for a kernel in one of the data structures based on the indicated number of completions accepted from the data structure. ### Claim 8 (depends on 1) The apparatus of claim 1, herein the distributed workload parser circuits include completion aggregator circuitry configured to determine the aggregate count in a single clock cycle using combinational logic. ### Claim 9 (depends on 1) The apparatus of claim 1, further comprising: counter circuitry configured to adjust a counter value based on the aggregate count. ### Claim 10 A method, comprising: maintaining, by a distributed workload parser circuit, for each of a set of shader processors, a data structure that specifies a count of workgroup completions for one or more kernels processed by the shader processor, wherein the distributed workload parser circuit is connected to a master workload parser circuit via a communications fabric; determining, by the distributed workload parser circuit, for the set of shader processors based on counts of workgroup completions for a first kernel, an aggregate count of completions to report for the first kernel; sending, by the distributed workload parser circuit, the aggregate count to the master workload parser circuit over the communications fabric; and adjusting, by the distributed workload parser circuit, the data structures to reflect counts included in the aggregate count. ### Claim 11 (depends on 10) The method of claim 10, wherein the data structure is a first-in first-out (FIFO) structure that includes an entry for each kernel for which a shader processor has completions and a corresponding completion count for that kernel. ### Claim 12 (depends on 11) The method of claim 11, further comprising: maintaining priority information for one or more of the FIFOs; wherein the determining is based on data in the oldest entry in each FIFO; and wherein the distributed workload parser circuit selects to report completions for the first kernel based on the first kernel being represented by the oldest entry in a shader FIFO having a current highest priority. ### Claim 13 (depends on 10) The method of claim 10, further comprising: receiving, by one or more aggregator circuits in a chain of aggregator circuits: a completion count from a corresponding data structure; an accumulated count from one or more prior aggregator circuits in the chain; and a kernel identifier of the kernel for which counts are being aggregated; and generating, by the one or more aggregator circuits: an updated accumulated count based on the received completion count and the received accumulated count; and a signal indicating a number of completions accepted from the corresponding data structure. ### Claim 14 (depends on 13) The method of claim 13, further comprising: determining a current highest-priority shader processor; and routing, by multiplexer circuitry, information from the data structure of the highest-priority shader to a starting aggregator circuit in the chain of aggregator circuits and routing the signals indicating the numbers of accepted completions to corresponding shader processors. ### Claim 15 A non-transitory computer readable storage medium having stored thereon design information that specifies a design of at least a portion of a hardware integrated circuit in a format recognized by a semiconductor fabrication system that is configured to use the design information to produce the circuit according to the design, wherein the design information specifies that the circuit includes: a plurality of shader processors configured to perform operations for compute workgroups included in compute kernels processed by the circuit; a master workload parser circuit; a plurality of distributed workload parser circuits; a communications fabric connected to the plurality of distributed workload parser circuits and the master workload parser circuit; where at least one of the distributed workload parser circuits is configured to: maintain, for each of a set of the shader processors, a data structure that specifies a count of workgroup completions for one or more kernels processed by the shader processor; determine, for the set of shader processors based on counts of workgroup completions for a first kernel, an aggregate count of completions to report for the first kernel; send the aggregate count to the master workload parser circuit over the communications fabric; and adjust the data structures to reflect counts included in the aggregate count. ### Claim 16 (depends on 15) The non-transitory computer readable storage medium of claim 15, wherein the data structure is a first-in first-out (FIFO) structure that includes an entry for each kernel for which a shader processor has completions and a corresponding completion count for that kernel. ### Claim 17 (depends on 16) The non-transitory computer readable storage medium of claim 16, wherein the distributed workload parser circuit is configured to determine the aggregate count based on data in the oldest entry in each FIFO; and wherein the circuit is configured to maintain priority information for one or more of the FIFOs and wherein the distributed workload parser circuit is configured to select to report completions for the first kernel based on the first kernel being represented by the oldest entry in a shader FIFO having a current highest priority. ### Claim 18 (depends on 15) The non-transitory computer readable storage medium of claim 15, wherein the distributed workload parser circuits include respective completion aggregator circuitry configured to determine the aggregate count, wherein the completion aggregator circuitry includes: a chain of aggregator circuits including one or more aggregator circuits configured to: receive: a completion count from a corresponding data structure; an accumulated count from one or more prior aggregator circuits in the chain; and a kernel identifier of the kernel for which counts are being aggregated; and generate: an updated accumulated count based on the received completion count and the received accumulated count; and a signal indicating a number of completions accepted from the corresponding data structure. ### Claim 19 (depends on 18) The non-transitory computer readable storage medium of claim 18, wherein the completion aggregator circuitry is configured to determine a current highest-priority shader processor; and wherein the completion aggregator circuitry further includes multiplexer circuitry configured to route information from the data structure of the highest-priority shader to a starting aggregator circuit in the chain of aggregator circuits and route the signals indicating the numbers of accepted completions to corresponding shader processors. ### Claim 20 (depends on 18) The non-transitory computer readable storage medium of claim 18, wherein the circuit is configured to adjust a count value for a kernel in one of the data structures based on the indicated number of completions accepted from the data structure.