Automatically labeling items using a machine-trained language model

A computing system for labeling items for use in training a machine-trained model, comprising:

• a memory;
• a processing system for executing computer-readable instructions, to perform operations including:
• in a language model interaction performed by a labeling computer control program, sending a prompt that includes a text-based task description and a particular item in a set of items to a machine-trained language model, the task description describing a group of classes to be used in labeling the particular item, the task description also providing an instruction to the language model to specify another class, not specified in the task description, upon determining that none of the classes specified in the task description applies to the particular item;
• as part of the language model interaction, receiving a response generated by the language model based on the task description and the particular item, the response specifying a particular class associated with the particular item;
• repeating, by the labeling computer control program, the language model interaction for other items in the set of items, to provide plural responses;
• determining, based on the responses, that an additional class, not among the classes specified in the task description, satisfies a prescribed test;
• automatically updating the group of classes in the task description in the prompt to include the additional class;
• producing, by the labeling control computer program, a training set that includes training examples, each training example including a given item and a ground-truth output result, the ground-truth output result being a class associated with the given item specified by the language model for the given item;
• and
• automatically training the machine-trained model based on the set of training examples, the automatically training involving iteratively updating weights of the machine-trained model.

A computer-readable storage medium for storing computer-readable instructions, a processing system executing the computer-readable instructions to perform operations, the operations comprising each of:

• sending plural text-based task descriptions to plural machine-trained language models in plural respective prompts, together with a particular item of a set of items,
• wherein a particular task description of the plural task descriptions describes a particular group of classes to be used to label the particular item, and wherein the particular task description also provides an instruction to the language model to specify another class, not specified in the particular task description, upon determining that none of the group of classes applies to the particular item, and
• wherein at least two task descriptions of the plural task descriptions differ from each other;
• receiving responses generated by the plural language models based on the plural task descriptions and the particular item, the responses including particular classes specified by the plural language models associated with the particular item;
• determining a final class for the particular item based on the particular classes generated by the plural language models;
• repeating the sending, receiving, and determining for other items in the set of items, to generate plural final classes for the items;
• producing a training set that includes training examples, each training example including a given item and a ground-truth output result, the ground-truth output result being a particular final class associated with the given item;
• and
• automatically training the machine-trained model based on the set of training examples, the automatically training involving iteratively updating weights of the machine-trained model.