Deep neural network optimization system for machine learning model scaling

Abstract

The present disclosure is related to techniques for optimizing artificial intelligence (AI) and/or machine learning (ML) models to reduce resource consumption while maintaining or improving AI/ML model performance. A sparse distillation framework (SDF) is provided for producing a class of parameter and compute efficient AI/ML models suitable for resource constrained applications. The SDF simultaneously distills knowledge from a compute heavy teacher model while also pruning a student model in a single pass of training, thereby reducing training and tuning times considerably. A self-attention mechanism may also replace CNNs or convolutional layers of a CNN to have better translational equivariance. Other embodiments may be described and/or claimed.

Dependencies are inferred from the claim text.