Systems and methods for providing flexible, multi-capacity models for use of deep neural networks in mobile devices

A persistent memory storage, having stored thereon a set of instructions that, when executed by a processor, cause the processor to:

• initiate a resource-aware runtime scheduler that encodes an inference accuracy and processing latency of each of a number of descendent models of a number of concurrently running applications into a cost function;
• provide the scheduler with dynamic information concerning runtime resources;
• and
• based on an output of the scheduler in response to the dynamic information, select a resource-accuracy trade-off scheme for a set of concurrently running deep learning models of the concurrently running applications, and determine a optimal amount of runtime resources to allocate to each model to jointly maximize an overall inference accuracy and minimize an overall processing latency of all the concurrently running applications.

A device comprising:

• a local memory of the device having at least one multi-capacity neural network stored thereon;
• at least one local processor operatively coupled to the memory;
• at least one input operatively connected to the processor;
• wherein the memory has stored thereon a set of instructions which, when executed by the processor, cause the processor to:
• provide at least one runtime computing resource condition to a resource-aware scheduler;
• run the resource-aware scheduler;
• determine from an output of the resource-aware scheduler whether to page in to a short-term memory of the local memory or page out to a long-term memory of the local memory a set of filters of the multi-capacity neural network based on the at least one runtime computing resource condition;
• and
• via the resource-aware scheduler, load from memory and execute a descendant of the multi-capacity neural network.

A software development system comprising:

• at least one server processor of a network in communication with a user computer;
• at least one memory in communication with the at least one server processor, having stored thereon a set of instructions which, when executed by the server processor, cause the server processor to:
• receive a request from the user computer;
• in response to the request from the user computer, obtain a deep learning model;
• transform the deep learning model into a multi-capacity deep learning model, wherein the multi-capacity model has an architecture allowing for adaptive model compression and recovery during runtime by paging in to short-term memory a first set of filters of the deep learning model or page out to long-term memory a second set of filters from the deep learning model based on a runtime computing resource condition;
• and
• send the multi-capacity deep learning model to the user computer.