-
- Downloads
Attention-Based Guided Structured Sparsity (GSS) (#51)
* Added GSS ("Attention-Based Guided Structured Sparsity of Deep Neural Networks") and an example for ResNet20 channel pruning. - The idea is to regularize the variance of the distribution of the parameter structures. Some structures will zero completely and the rest should have a high value leading to a high variance. - A new regularizer class, GroupVarianceRegularizer, is used to regularize the group variance (effectively rewarding the loss function for high variance between the groups). - When tested on ResNet 20 GSS did not show any improvement over SSL * Added sample of filter pruning for ResNet20 CIFAR using SSL (Learning Structured Sparsity in Deep Neural Networks) * Added an example of pruning 45% of the compute (1.8x MAC reduction), while suffering 0.8% accuracy loss on ResNet20 CIFAR * Added a ResNet50 ImageNet example of L1-Magnitude fine-grained pruning, using an AGP schedule: 46% sparsity with a 0.6% accuracy increase. This is an example of using pruning used as a regularizer.
Showing
- distiller/config.py 2 additions, 2 deletionsdistiller/config.py
- distiller/regularization/__init__.py 1 addition, 1 deletiondistiller/regularization/__init__.py
- distiller/regularization/group_regularizer.py 70 additions, 31 deletionsdistiller/regularization/group_regularizer.py
- distiller/thresholding.py 1 addition, 1 deletiondistiller/thresholding.py
- examples/gss/gss_channels-removal_training.yaml 82 additions, 0 deletionsexamples/gss/gss_channels-removal_training.yaml
- examples/ssl/ssl_channels-removal_training.yaml 1 addition, 1 deletionexamples/ssl/ssl_channels-removal_training.yaml
- examples/ssl/ssl_channels-removal_training_x1.8.yaml 4 additions, 1 deletionexamples/ssl/ssl_channels-removal_training_x1.8.yaml
- examples/ssl/ssl_filter-removal_training.yaml 101 additions, 0 deletionsexamples/ssl/ssl_filter-removal_training.yaml
Loading
Please register or sign in to comment