Out of order memory request tracking structure and technique

A streaming cache configured to receive memory requests for data in a memory system and return memory system data in response to the received memory requests, the streaming cache comprising:

• a tag pipeline including a coalescer, a tag memory and a tag processor, the tag pipeline detecting cache misses;
• a commit FIFO;
• an evict FIFO;
• an out-of-order memory request tracking circuit comprising plural tracking queues each configured to maintain miss traffic in first-in-first-out order, the out-of-order memory request tracking circuit configured to distribute queueing of memory request tracking entries for out-of-order memory requests across to different tracking queues and to distributing queuing of memory request tracking entries for in-order memory requests to the same tracking queue;
• and
• a pipelined picker circuit connected to the plural tracking queues, the pipelined picker circuit configured to pick between multiple ones of the plural tracking queues having ready returns and enqueuing a picked ready return in a further first-in-first-out queue for delivery to a requester.

A memory request tracking method for use with a streaming cache memory configured to receive memory requests for data in a memory system and return memory system data in response to the received memory requests, the memory request tracking method comprising:

• detecting cache misses;
• providing plural tracking queues each configured to maintain miss traffic in first-in-first-out order;
• and
• distributing request tracking information entries for miss traffic to the plural tracking queues to support either in-order and out-of-order memory request returns including distributing queueing for out-of-order memory requests across different tracking queues and distributing queueing for in-order memory requests to the same tracking queue,
• wherein the distributing includes distributing queuing for the out-of-order memory request returns across plural tracking queues to reduce the chance that any individual long-latency access will block a number of other accesses, thereby enabling a consuming ray tracer to make forward progress when individual long-latency accesses occur.