Reducing queue synchronization of multiple work items in a system with high memory latency between processing nodes

A data processing system comprising:

• a global command queue (GCQ) that holds work elements that are processed via multiple processing units across multiple processor nodes, wherein the GCQ is accessible via work requests generated by one or more execution units;
• a plurality of processing nodes coupled via a system interconnect fabric to a system memory holding the GCQ, each processing node having one or more of the execution units, and wherein at least a first processing node exhibits a high access latency to the GCQ;
• work scheduling logic at the first processor node that configures the data processing system to:
• establish a local command/work queue (LCQ) in which multiple work items for execution by local processing units are staged prior to execution;
• cause a first local processing unit to generate a work request for retrieval of work from the GCQ;
• retrieve, via the work request, a larger chunk size of work than can be completed in a normal work completion/execution cycle by the local processing unit, wherein the larger chunk size is larger than a standard chunk size that is retrieved when the processing node is a low latency processing node;
• store the larger chunk size of work retrieved in a local command/work queue (LCQ);
• enable the first local processing unit to locally schedule and complete portions of the work stored within the LCQ;
• and
• generate a next work request for transmitting to the GCQ only when all the work within the LCQ has been dispatched by at least the first local processing unit;
• determine, in real time, a consumption speed at which the high-latency processing node consumes and completes the larger chunk size of work retrieved from the GCQ;
• and
• dynamically modify a local chunk size based on one or more factors, including the consumption speed, wherein said dynamically modifying the local chunk size includes:
• in response to the consumption speed indicating one or more local processing units will become idle during a request cycle, increase the local chunk size to request a larger chunk size within a next work request issued from the high-latency processing node, wherein a periodicity of issuing work requests from the high-latency processing node to the GCQ is reduced by increasing the local chunk size to accommodate the consumption speed of the local processing units;
• and
• reduce the local chunk size when the consumption speed indicates that the local processing units are not dispatching the larger chunk size timely, relative to the dispatching of similar amounts of work at other processing nodes.