Optimizing machine learning model performance

A processing system, comprising:

• a memory comprising computer-executable instructions;
• and
• a processor configured to execute the computer-executable instructions and cause the processing system to:
• receive data defining a neural network;
• analyze the data to determine a depth-first cut point for a depth-first traversal portion of an overall network traversal;
• perform depth-first traversal for the depth-first traversal portion of the overall network traversal;
• and
• perform layer-based traversal for a layer-based portion of the overall network traversal.

A method, comprising:

• receiving data defining a neural network;
• receiving hardware characteristic data defining a plurality of hardware resources, including a local memory capacity;
• analyzing the data to determine a cut point for a depth-first traversal portion of an overall network traversal;
• determining a first cut point based at least in part on the data and the local memory capacity, wherein the first cut point is at a maximum depth of the neural network given the local memory capacity;
• determining a second cut point, wherein the second cut point is shallower than the first cut point, and wherein the second cut point increases hardware parallelism compared to the first cut point;
• and
• performing depth-first traversal for the depth-first portion of the overall network traversal using two or more hardware resources of the plurality of hardware resource and the second cut point.

A method, comprising:

• performing, during a first time interval, a computational phase of a first inference on a first hardware resource;
• performing, during a second time interval, a transport phase of the first inference on a second hardware resource;
• performing, during the second time interval, a computational phase of a second inference on the first hardware resource;
• and
• performing, during a third time interval, a transport phase of the second inference on the second hardware resource.