Systems and methods for building artificial intelligence agents

A system for training a neural network based language model (LM), the system comprising:

• a memory that stores the LM and a plurality of processor executable instructions;
• a communication interface that receives a training dataset including pairs of user queries and ground-truth response;
• and
• one or more hardware processors that read and execute the plurality of processor-executable instructions from the memory to perform operations comprising:
• generating, via the LM, a plurality of responses based on a query from the training dataset;
• identifying, from the plurality of responses, a first response having a first similarity metric value below a threshold, based on a similarity metric associated with a corresponding ground-truth response from the training dataset;
• training the LM using the first response as a negative sample and a second response as a positive sample such that the LM after training is more likely to generate the positive sample and less likely to generate the negative sample;
• receiving, via a user interface, a user query;
• and
• generating a response to the user query via the trained LM.