Constraining output of a generative language model to conform to a grammar

A system comprising:

• a memory to store a grammar defining valid sequences of output;
• and
• at least one processing unit to:
• obtain a plurality of values generated using a generative language model, each of the values indicative of a probability of a respective token being a next token in a token sequence generated by the generative language model;
• obtain a mask based on at least one token of the token sequence already generated by the generative language model and based on the grammar defining the valid sequences of output;
• and
• apply the mask to the plurality of values, the mask operating on each value that corresponds to a token not compliant with the grammar to reduce or zero the probability of the corresponding token being the next token;
• wherein the next token is determined based on the plurality of values after the mask is applied.

One or more non-transitory computer-readable storage media having stored thereon computer-executable instructions that, when executed by at least one processing unit, cause the at least one processing unit to perform operations comprising:

• obtaining a plurality of values generated using a generative language model, each of the values indicative of a probability of a respective token being a next token in a token sequence generated by the generative language model;
• obtaining a mask based on at least one token of the token sequence already generated by the generative language model and based on a grammar defining valid sequences of output;
• and
• applying the mask to the plurality of values, the mask operating on each value that corresponds to a token not compliant with the grammar to reduce or zero the probability of the corresponding token being the next token;
• wherein the next token is determined based on the plurality of values after the mask is applied.