Execution of segmented machine learning models

A device, comprising:

• a first type of memory;
• a second type of memory that differs from the first type of memory;
• and
• at least one processor configured to:
• receive a request to provide an input to one or more machine learning (ML) models comprising layers, the one or more ML models stored in the first type of memory;
• divide the layers of the one or more ML models into a plurality of segments such that at least one of the plurality of segments fits within allocated space of the second type of memory;
• load a first segment of the plurality of segments from the first type of memory into the second type of memory;
• cause the input to be processed through the first segment of the plurality of segments using the second type of memory while a second segment of the plurality of segments is loaded from the first type of memory into the second type of memory;
• and
• provide an output of the one or more machine learning models after performing the loading and processing for at least some of the plurality of segments.

A computer program product comprising code, stored in a non-transitory computer-readable storage medium, the code comprising:

• code to receive, by a first device, a request to provide an input to one or more machine learning (ML) models comprising layers, the one or more ML models stored in a first type of memory;
• code to divide, by the first device, the layers of the one or more machine learning models into a plurality of segments such that at least one of the plurality of segments fits within allocated space of a second type of memory that differs from the first type of memory;
• code to load, by the first device, a first segment of the plurality of segments from the first type of memory into the second type of memory;
• code to cause, by the first device, the input to be processed through the first segment of the plurality of segments using the second type of memory while a second segment of the plurality of segments is loaded from the first type of memory into the second type of memory;
• and
• code to provide, by the first device, an output of the one or more machine learning models after performing the loading and processing for at least some of the plurality of segments.