Fault tolerant logging for artificial intelligence applications

A method comprising:

• executing, by a processor, a worker on a server host, wherein the worker comprises a log messenger and at least one processing thread;
• determining, by the processor, a deployment location for a message broker from among the server host and a remote host based on at least one predicted logging attribute of an artificial intelligence (AI) pipeline executed by the worker;
• executing, by the processor, the message broker at the deployment location;
• executing, by the processor, at least one artificial intelligence (AI) pipeline via the at least one processing thread of the worker to generate log messages;
• storing, by the processor, the log messages within a queue of the message broker;
• and
• in response to a determination that a log message has reached an end of the queue, writing, by the message broker, the log message to a log database.

A non-transitory computer-readable storage medium comprising instructions which when executed by a computer cause a processor to perform:

• executing a worker on a server host, wherein the worker comprises a log messenger and at least one processing thread;
• determining a deployment location for a message broker from among the server host and a remote host based on at least one predicted logging attribute of an artificial intelligence (AI) pipeline executed by the worker;
• executing the message broker at the deployment location;
• executing the AI pipeline via the at least one processing thread of the worker to generate log messages;
• transferring the log messages from the log messenger to the message broker;
• storing the log messages within a queue of the message broker;
• and
• in response to a determination that a log message has reached an end of the queue, writing, by the message broker, the log message to a log database.