Systems and methods near negative distinction for evaluating NLP models

A system for evaluating a model, the system comprising:

• a communication interface receiving an evaluation dataset comprising a plurality of unit tests, the unit tests having:
• an input context, and
• a first candidate and a second candidate that are generated in response to the input context, wherein the first candidate is associated with a first quality notation, and the second candidate is associated with a second quality notation;
• a memory storing a plurality of processor-executable instructions;
• and
• a processor reading and executing the instructions from the memory to perform operations comprising:
• determine, via a model, a first likelihood of generating the first candidate in response to the input context;
• determine, via the model, a second likelihood of generating the second candidate in response to the input context;
• determining a count of unit tests in the evaluation dataset where the model passed;
• determining an aggregate pass rate based on the determined count of unit tests, the aggregate pass rate indicating whether the model produces candidates that correspond to the first quality notation;
• determine whether the first likelihood being greater than the second likelihood base on the aggregate pass rate;
• and
• based on a determination that the first likelihood is greater than the second likelihood, determining whether the model passed a unit test in the evaluation dataset, wherein the first quality notation is associated with a higher quality candidate and the second quality notation is associated with a lower quality candidate.

A processor-readable non-transitory storage medium storing a plurality of processor-executable instructions for knowledge base question answering, the instructions being executed by one or more processors to perform operations comprising:

• receiving an evaluation dataset comprising a plurality of unit tests, the unit tests having:
• an input context, and
• a first candidate and a second candidate that are generated in response to the input context, wherein the first candidate is associated with a first quality notation, and the second candidate is associated with a second quality notation;
• determining, via a model, a first likelihood of generating the first candidate in response to the input context;
• determining, via the model, a second likelihood of generating the second candidate in response to the input context;
• determining a count of unit tests in the evaluation dataset where the model passed;
• determining an aggregate pass rate based on the determined count of unit tests, the aggregate pass rate indicating whether the model produces candidates that correspond to the first quality notation;
• determining whether the first likelihood being greater than the second likelihood based on the aggregate pass rate;
• and
• based on a determination that the first likelihood is greater than the second likelihood, determining whether the model passed one unit test in the evaluation dataset, wherein the first quality notation is associated with a higher quality candidate and the second quality notation is associated with a lower quality candidate.