US 2026/0093985 A1 · Training Transformers Using Sliceout

Abstract

A system for training the neural network using dropout with slicing operations preserves the regularization effects of dropout, while speeding up computations and reducing the memory requirements of training the neural network. Instead of randomly dropping weights connected to neurons in a neural network, the system slices contiguous memory segments of weight matrices. For transformer models, the approach first receives input data that consist of a sequence of elements. Based on the input data, input embedding vectors with positional encoding are generated. Then the transformer model is trained by passing the input embedding vectors through various neural network layers. While passing through linear layers, some of the weight matrices are sliced (e.g., masked) such that a contiguous section of a weight matrix is kept unsliced and used for training and the rest of the weight matrix is not accessed.

Claims20

Dependent claims

A method for training a transformer model comprising:

receiving input data, the input data containing a sequence of elements;
generating embedded input vectors based on the input data;
generating a plurality of matrices based on the embedded input vectors;
applying a mask to a matrix of the plurality of matrices, the mask selecting a contiguous section of the matrix resulting in a sliced matrix;
and
training the transformer model based at least in part on the sliced matrix by:
training the transformer model with a first subset of data of the input data using a first mask set;
and
training the transformer model with a second subset of data of the input data using a second mask set, wherein the second subset of data is different from the first subset of data and the second mask set is different from the first mask set.

6 more elements

Dependencies are inferred from the claim text.

Training Transformers Using Sliceout

Life of this patent

Training Transformers Using Sliceout

Life of this patent

Abstract

Claims20