Use of privileged information to improve automatic evaluations

A system comprising:

• one or more computers;
• and
• one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
• obtaining a first solution generated by a candidate model using a first query;
• generating a performance metric, using an evaluation model and conditioned on the first query, first solution and privileged information, wherein the performance metric comprises an evaluation output representing a performance of the candidate model;
• wherein the privileged information was not available to the candidate model when generating the first solution