Executing large artificial intelligence models on memory-constrained devices

At least one computer-readable storage medium having program instructions recorded thereon that, when executed by at least one processing circuit of a group of target devices, configure a first target device of the group of target devices to perform a method, each target device of the group of target devices being communicatively connected to a parameter server that stores a master copy of an artificial intelligence (AI) model, the group of target devices being configured to run an instance of the AI model, the first target device comprising an integrated circuit chip having an on-chip memory, a downloader, a data manager, and an executer, the method comprising:

• downloading, by the downloader, a portion of the AI model from the parameter server, a size of the portion being based at least on a size of the on-chip memory and a size of one or more layers of the AI model;
• storing, by the data manager, a set of microbatches of a dataset;
• executing, by the executer and via the integrated circuit chip, the set of microbatches on a first subportion of the downloaded portion of the AI model stored in the on-chip memory to generate activations;
• and
• downloading, by the downloader, weights for a second subportion of the downloaded portion of the AI from the parameter server.

A target device comprising an integrated circuit chip having an on-chip memory of a size less than an entirety of an artificial intelligence (AI) model, the target device configured to:

• download a portion of the AI model from a parameter server, a size of the portion being based at least on the on-chip memory size and a size of one or more layers of the AI model;
• store a set of microbatches of a dataset;
• execute by the integrated circuit chip at the target device the set of microbatches on a first subportion of the downloaded portion of the AI model stored in the on-chip memory to generate activations;
• and
• download weights for a second subportion of the downloaded portion of the AI model from the parameter server.

A computer-readable storage medium having program instructions recorded thereon that, when executed by at least one processing circuit of a target device, configure the target device to perform a method that comprises:

• downloading a portion of an artificial intelligence (AI) model from a parameter server, the target device comprising an integrated circuit chip having an on-chip memory, and a size of the portion being based at least on a size of the on-chip memory and a size of one or more layers of the AI model;
• storing a set of microbatches of a dataset;
• executing by the integrated circuit chip at the target device the set of microbatches on a first subportion of the downloaded portion of the AI model stored in the on-chip memory to generate activations;
• and
• downloading weights for a second subportion of the downloaded portion of the AI model from the parameter server.