Constrained masking for sparsification in machine learning
AssigneeQUALCOMM Incorporated
InventorsDebasmit Das, Jamie Menjay Lin
Life of this patent
Estimating the term…
Abstract
Certain aspects of the present disclosure provide techniques and apparatus for feature masking. A feature tensor is accessed in a neural network, and a feature mask is generated by processing the feature tensor using a masking subnetwork, where the masking subnetwork was trained based at least in part on a polarization constraint and an activation constraint to generate feature masks. A masked feature tensor is generated based on the feature tensor and the feature mask, and an output inference is generated using the neural network based at least in part on the masked feature tensor.
Claims24
A processor-implemented method of machine learning, comprising:
- accessing a feature tensor in a neural network;
- generating a feature mask by processing the feature tensor using a masking subnetwork, wherein the masking subnetwork was trained based at least in part on a polarization constraint and an activation constraint to generate feature masks;
- generating a polarization loss based at least in part on the feature mask, wherein the polarization loss is generated based at least in part on an expectation of a product of the feature mask and an inverse of the feature mask;
- refining one or more parameters of the masking subnetwork based on the polarization loss;
- generating a masked feature tensor based on the feature tensor and the feature mask;
- and
- generating an output inference using the neural network based at least in part on the masked feature tensor.
A system comprising:
- memory comprising computer-executable instructions;
- and
- one or more processors configured to execute the computer-executable instructions and cause the system to perform an operation comprising:
- accessing a feature tensor in a neural network;
- generating a feature mask by processing the feature tensor using a masking subnetwork, wherein the masking subnetwork was trained based at least in part on a polarization constraint and an activation constraint to generate feature masks;
- generating a polarization loss based at least in part on the feature mask, wherein the polarization loss is generated based at least in part on an expectation of a product of the feature mask and an inverse of the feature mask;
- refining one or more parameters of the masking subnetwork based on the polarization loss;
- generating a masked feature tensor based on the feature tensor and the feature mask;
- and
- generating an output inference using the neural network based at least in part on the masked feature tensor.
One or more non-transitory computer-readable media comprising instructions that, when executed by one or more processors of a processing system, cause the processing system to perform an operation comprising:
- accessing a feature tensor in a neural network;
- generating a feature mask by processing the feature tensor using a masking subnetwork, wherein the masking subnetwork was trained based at least in part on a polarization constraint and an activation constraint to generate feature masks;
- generating a polarization loss based at least in part on the feature mask, wherein the polarization loss is generated based at least in part on an expectation of a product of the feature mask and an inverse of the feature mask;
- refining one or more parameters of the masking subnetwork based on the polarization loss;
- generating a masked feature tensor based on the feature tensor and the feature mask;
- and
- generating an output inference using the neural network based at least in part on the masked feature tensor.
A system comprising:
- means for accessing a feature tensor in a neural network;
- means for generating a feature mask by processing the feature tensor using a masking subnetwork, wherein the masking subnetwork was trained based at least in part on a polarization constraint and an activation constraint to generate feature masks;
- means for generating a polarization loss based at least in part on the feature mask, wherein the polarization loss is generated based at least in part on an expectation of a product of the feature mask and an inverse of the feature mask;
- means for refining one or more parameters of the masking subnetwork based on the polarization loss;
- means for generating a masked feature tensor based on the feature tensor and the feature mask;
- and
- means for generating an output inference using the neural network based at least in part on the masked feature tensor.
Dependencies are inferred from the claim text.