Inference model on restrained gpu memory

A non-transitory computer-readable storage media that configures a computer to perform program instructions stored on the non-transitory computer-readable storage media for running an inference model with a graphical processing unit (GPU) having a restrained resource, the program instructions comprising:

• receiving an input for which the inference model is run using a sequential inference process comprising a plurality of steps, the inference model comprising a plurality of layers including a first layer representing the input, a last layer representing an output for the input as generated by running the inference model, and at least one intermediate layer between the first layer and the last layer;
• determining information associated with the inference model including an input size for each of the layers, an output size for each of the layers, and an available GPU memory on which the inference model is to run, the available GPU memory being the restrained resource;
• determining a count (M) of the layers based on the information associated with the inference model, the count being a value less than a total number of the layers of the inference model, each of the steps of the sequential inference process loading and running M layers of the layers of the inference model;
• determining allocations in the available GPU memory for each of the steps of the sequential inference process, the allocations configured for data associated with the M layers, a step input and a step output, and intermediate information, the step input being input data for one of the steps, the step output being output data for the one of the steps, and the intermediate information being memory identification information for the step output;
• loading and running the M layers using the step input to calculate the step output;
• and
• generating the intermediate information for the step output for a subsequent step to utilize the step output as a further step input in the subsequent step.

A computer system for running an inference model with a graphical processing unit (GPU) having a restrained resource, the computer system comprising:

• one or more computer processors, one or more computer-readable storage media, and program instructions stored on the one or more of the computer-readable storage media for execution by at least one of the one or more processors capable of performing a method, the method comprising:
• receiving an input for which the inference model is run using a sequential inference process comprising a plurality of steps, the inference model comprising a plurality of layers including a first layer representing the input, a last layer representing an output for the input as generated by running the inference model, and at least one intermediate layer between the first layer and the last layer;
• determining information associated with the inference model including an input size for each of the layers, an output size for each of the layers, and an available GPU memory on which the inference model is to run, the available GPU memory being the restrained resource;
• determining a count (M) of the layers based on the information associated with the inference model, the count being a value less than a total number of the layers of the inference model, each of the steps of the sequential inference process loading and running M layers of the layers of the inference model;
• determining allocations in the available GPU memory for each of the steps of the sequential inference process, the allocations configured for data associated with the M layers, a step input and a step output, and intermediate information, the step input being input data for one of the steps, the step output being output data for the one of the steps, and the intermediate information being memory identification information for the step output;
• loading and running the M layers using the step input to calculate the step output;
• and
• generating the intermediate information for the step output for a subsequent step to utilize the step output as a further step input in the subsequent step.