Inference with inline real-time ML models in applications

An apparatus for executing a machine-learning model within an application, comprising:

• a memory storing computer-executable instructions;
• and
• at least one processor configured to execute the instructions to:
• receive compiled and verified codelet, wherein extended Berkeley packet filter (eBPF) bytecode of the codelet has been verified to satisfy safety requirements for execution within the application, wherein the codelet includes code implementing a pre-trained machine-learning model;
• load the verified codelet into a hook point of the application, wherein to load the verified codelet into the hook point of the application, the at least one processor is configured to:
• receive a configuration map for the codelet while the application is executing in a main container;
• mount code including a function associated with the hook point of the application to the main container;
• and
• make a location that contains the verified codelet available to the container;
• and
• execute the verified codelet to apply application data to the machine-learning model to infer a result;
• add a component to the container configured to monitor whether to execute a codelet at the hook point and to select which codelet to execute at the hook point;
• detect an annotation for the main container that identifies the verified codelet during execution of the application;
• and
• symbolically link the verified codelet indicated by the annotation to the hook point of the application.