Drop-layer based vector augmentation

A system comprising:

• at least one data processor;
• and
• at least one memory storing instructions, which when executed by the at least one data processor, cause operations comprising:
• generating vectors comprising a plurality of tokens representative of a plurality of sentences, the plurality of sentences including text that is included as part of training data;
• performing an augmentation operation on a first vector of the vectors and a second vector of the vectors, the augmentation operation comprising:
• merging, in accordance with a weighting factor, the first vector with the second vector such that at least a first token of the first vector is combined with a second token of the second vector, wherein the first token is representative of a first sentence of the plurality of sentences and the second token represents a second sentence of the plurality of sentences;
• inputting, the first vector that is merged with the second vector, into a first encoding layer in a transformer stack, the first encoding layer being configured to receive a vector as input and convert the vector into a vector with a different format;
• outputting, by the first encoding layer, a third vector that is based on the first vector that is merged with the second vector, the third vector including a third token that includes a combination of the first sentence and the second sentence;
• and
• generating a modified version of the training data comprising the third vector for training a machine learning model.

At least one non-transitory computer readable media storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

• generating vectors comprising a plurality of tokens representative of a plurality of sentences, the plurality of sentences including text that is included as part of training data;
• performing an augmentation operation on a first vector of the vectors and a second vector of the vectors, the augmentation operation comprising:
• merging, in accordance with a weighting factor, the first vector with the second vector such that at least a first token of the first vector is combined with a second token of the second vector, wherein the first token is representative of a first sentence of the plurality of sentences and the second token represents a second sentence of the plurality of sentences;
• inputting, the first vector that is merged with the second vector, into a first encoding layer in a transformer stack, the first encoding layer being configured to receive a vector as input and convert the vector into a vector with a different format;
• outputting, by the first encoding layer, a third vector that is based on the first vector that is merged with the second vector, the third vector including a third token that includes a combination of the first sentence and the second sentence;
• and
• generating a modified version of the training data comprising the third vector for training the machine learning model.