Model parameter sharing between inference application instances in processing unit of information processing system

A method, comprising:

• obtaining, via a first process, an inference model for which multiple instances of the inference model are to be executed on at least one processing unit;
• differentiating, using a host manager, from the inference model, via the first process, one or more model parameters that are a pre-trained type of model parameter, the differentiating comprising:
• parsing the inference model;
• generating a computation table for each computation of a plurality of computations defined in the inference model, the computation table comprising a plurality of computation nodes, the computation nodes comprising computation node numbers;
• generating a parameter table for each parameter used by the plurality of computations, the parameter table comprising a plurality of parameter nodes, the parameter nodes comprising parameter node numbers, wherein each computation of the plurality of computations is indicated by a computation node number of the computation table and one or more parameter node numbers of the parameter table;
• generating a memory model and associating the computation table and the parameter table using the generated memory model to determine one or more parameter node numbers of the parameter table associated with each computation node number of the plurality of computations, and wherein one or more parameter nodes of the plurality of parameter nodes are associated with two or more computation nodes of the plurality of computation nodes;
• inferring a shape of each parameter node of each computation;
• and
• identifying the one or more model parameters that are the pre-trained type of model parameter;
• determining, using the host manager, based on the inferred shape of the parameter nodes that are identified as the one or more model parameters that are the pre-trained type of model parameter, an amount of memory to store the one or more model parameters that are the pre-trained type of model parameter in a shared memory storage space;
• allocating, via the first process, a shared memory of the shared memory storage space corresponding to the amount of memory for storing the one or more model parameters that are the pre-trained type of model parameter;
• extracting, using the host manager, the one or more model parameters that are the pre-trained type of model parameter from the inference model;
• storing, via the first process, the one or more model parameters that are the pre-trained type of model parameter in the shared memory for access by the multiple instances of the inference model to be executed on the at least one processing unit;
• and
• permitting access to the shared memory of the shared memory storage space created by the first process to allow the multiple instances of the inference model to utilize the one or more model parameters that are the pre-trained type of model parameter in the shared memory of the shared memory storage space while executing the inference model.

A computer program product stored on a non-transitory computer-readable medium and comprising machine executable instructions, the machine executable instructions, when executed, causing a processing platform to:

• obtain, via a first process, an inference model for which multiple instances of the inference model are to be executed on at least one processing unit;
• differentiate, using a host manager, from the inference model, via the first process, one or more model parameters that are a pre-trained type of model parameter, the differentiating comprising:
• parsing the inference model;
• generating a computation table for each computation of a plurality of computations defined in the inference model, the computation table comprising a plurality of computation nodes, the computation nodes comprising computation node numbers;
• generating a parameter table for each parameter used by the plurality of computations, the parameter table comprising a plurality of parameter nodes, the parameter nodes comprising parameter node numbers, wherein each computation of the plurality of computations is indicated by a computation node number of the computation table and one or more parameter node numbers of the parameter table;
• generating a memory model and associating the computation table and the parameter table using the generated memory model to determine one or more parameter node numbers of the parameter table associated with each computation node number of the plurality of computations, and wherein one or more parameter nodes of the plurality of parameter nodes are associated with two or more computation nodes of the plurality of computation nodes;
• inferring a shape of each parameter node of each computation;
• and
• identifying the one or more model parameters that are the pre-trained type of model parameter;
• determine, using the host manager, based on the inferred shape of the parameter nodes that are identified as the one or more model parameters that are the pre-trained type of model parameter, an amount of memory to store the one or more model parameters that are the pre-trained type of model parameter in a shared memory storage space;
• allocate, via the first process, a shared memory of the shared memory storage space corresponding to the amount of memory for storing the one or more model parameters that are the pre-trained type of model parameter;
• extract, using the host manager, the one or more model parameters that are the pre-trained type of model parameter from the inference model;
• store, via the first process, the one or more model parameters that are the pre-trained type of model parameter in the shared memory for access by the multiple instances of the inference model to be executed on the at least one processing unit;
• and
• permit access to the shared memory of the shared memory storage space created by the first process to allow the multiple instances of the inference model to utilize the one or more model parameters that are the pre-trained type of model parameter in the shared memory of the shared memory storage space while executing the inference model.