Content adaptive attention model for neural network-based image and video encoders
AssigneeIntel Corporation
InventorMan Lan Wong
Life of this patent
Estimating the term…
Abstract
Various embodiments are generally directed to using attention models in neural network-based image and video encoders and/or decoders. A first feature map of a first image may be generated by a first layer of a neural network, the neural network executing on a computer processor to encode the first image. An attention layer of the neural network may compute an adaptive spatial saliency map for the first feature map of the first image based on the first feature map of the first image. The neural network may then perform an element-wise multiplication of the first feature map and the adaptive spatial saliency map for the first feature map to generate a modulated feature map to encode the first image.
Claims25
An apparatus, comprising:
- a processor circuit;
- and
- a memory storing instructions which when executed by the processor circuit cause the processor circuit to:
- generate a first feature map of a first image by a first layer of a neural network, the neural network to encode the first image at a defined bitrate;
- compute, by an attention layer of the neural network based on the first feature map of the first image, an adaptive spatial saliency map for the first feature map of the first image, wherein a sum of all values of the adaptive spatial saliency map equals one;
- and
- perform an element-wise multiplication of the first feature map and the adaptive spatial saliency map for the first feature map to generate a modulated feature map to encode the first image, the modulated feature map to have relatively higher activations on portions of the first image that are higher regions of interest to a human visual system (HVS) and relatively lower activations on portions of the first image that are lower regions of interest to the HVS while keeping the defined bitrate constant.
A non-transitory computer-readable storage medium comprising instructions that when executed by a processor of a computing device, cause the processor to:
- generate a first feature map of a first image by a first layer of a neural network, the neural network to encode the first image at a defined bitrate;
- compute, by an attention layer of the neural network based on the first feature map of the first image, an adaptive spatial saliency map for the first feature map of the first image, wherein a sum of all values of the adaptive spatial saliency map equals one;
- and
- perform an element-wise multiplication of the first feature map and the adaptive spatial saliency map for the first feature map to generate a modulated feature map to encode the first image, the modulated feature map to have relatively higher activations on portions of the first image that are higher regions of interest to a human visual system (HVS) and relatively lower activations on portions of the first image that are lower regions of interest to the HVS while keeping the defined bitrate constant.
A method, comprising:
- generating a first feature map of a first image by a first layer of a neural network, the neural network executing on a computer processor to encode the first image at a defined bitrate;
- computing, by an attention layer of the neural network based on the first feature map of the first image, an adaptive spatial saliency map for the first feature map of the first image, wherein a sum of all values of the adaptive spatial saliency map equals one;
- and
- performing an element-wise multiplication of the first feature map and the adaptive spatial saliency map for the first feature map to generate a modulated feature map to encode the first image, the modulated feature map to have relatively higher activations on portions of the first image that are higher regions of interest to a human visual system (HVS) and relatively lower activations on portions of the first image that are lower regions of interest to the HVS while keeping the defined bitrate constant.
Dependencies are inferred from the claim text.