-
- Downloads
Activation statistics collection (#61)
Activation statistics can be leveraged to make pruning and quantization decisions, and so We added support to collect these data. - Two types of activation statistics are supported: summary statistics, and detailed records per activation. Currently we support the following summaries: - Average activation sparsity, per layer - Average L1-norm for each activation channel, per layer - Average sparsity for each activation channel, per layer For the detailed records we collect some statistics per activation and store it in a record. Using this collection method generates more detailed data, but consumes more time, so Beware. * You can collect activation data for the different training phases: training/validation/test. * You can access the data directly from each module that you chose to collect stats for. * You can also create an Excel workbook with the stats. To demonstrate use of activation collection we added a sample schedule which prunes weight filters by the activation APoZ according to: "Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures", Hengyuan Hu, Rui Peng, Yu-Wing Tai, Chi-Keung Tang, ICLR 2016 https://arxiv.org/abs/1607.03250 We also refactored the AGP code (AutomatedGradualPruner) to support structure pruning, and specifically we separated the AGP schedule from the filter pruning criterion. We added examples of ranking filter importance based on activation APoZ (ActivationAPoZRankedFilterPruner), random (RandomRankedFilterPruner), filter gradients (GradientRankedFilterPruner), and filter L1-norm (L1RankedStructureParameterPruner)
Showing
- distiller/__init__.py 4 additions, 8 deletionsdistiller/__init__.py
- distiller/data_loggers/__init__.py 1 addition, 1 deletiondistiller/data_loggers/__init__.py
- distiller/data_loggers/collector.py 265 additions, 51 deletionsdistiller/data_loggers/collector.py
- distiller/data_loggers/logger.py 21 additions, 13 deletionsdistiller/data_loggers/logger.py
- distiller/data_loggers/tbbackend.py 33 additions, 10 deletionsdistiller/data_loggers/tbbackend.py
- distiller/policy.py 1 addition, 0 deletionsdistiller/policy.py
- distiller/pruning/__init__.py 5 additions, 2 deletionsdistiller/pruning/__init__.py
- distiller/pruning/automated_gradual_pruner.py 44 additions, 16 deletionsdistiller/pruning/automated_gradual_pruner.py
- distiller/pruning/ranked_structures_pruner.py 176 additions, 11 deletionsdistiller/pruning/ranked_structures_pruner.py
- distiller/thresholding.py 96 additions, 86 deletionsdistiller/thresholding.py
- distiller/utils.py 0 additions, 18 deletionsdistiller/utils.py
- examples/agp-pruning/resnet20_filters.schedule_agp.yaml 66 additions, 32 deletionsexamples/agp-pruning/resnet20_filters.schedule_agp.yaml
- examples/agp-pruning/resnet20_filters.schedule_agp_2.yaml 46 additions, 34 deletionsexamples/agp-pruning/resnet20_filters.schedule_agp_2.yaml
- examples/classifier_compression/compress_classifier.py 74 additions, 23 deletionsexamples/classifier_compression/compress_classifier.py
- examples/network_trimming/resnet56_cifar_activation_apoz.yaml 212 additions, 0 deletions...ples/network_trimming/resnet56_cifar_activation_apoz.yaml
- examples/network_trimming/resnet56_cifar_activation_apoz_v2.yaml 198 additions, 0 deletions...s/network_trimming/resnet56_cifar_activation_apoz_v2.yaml
- examples/pruning_filters_for_efficient_convnets/resnet56_cifar_filter_rank.yaml 14 additions, 6 deletions...rs_for_efficient_convnets/resnet56_cifar_filter_rank.yaml
- examples/pruning_filters_for_efficient_convnets/resnet56_cifar_filter_rank_v2.yaml 207 additions, 0 deletions...for_efficient_convnets/resnet56_cifar_filter_rank_v2.yaml
- examples/sensitivity-analysis/resnet50-imagenet/resnet50.imagenet.sensitivity_filter_wise.csv 584 additions, 0 deletions...50-imagenet/resnet50.imagenet.sensitivity_filter_wise.csv
- jupyter/imagenet_classes.py 1003 additions, 0 deletionsjupyter/imagenet_classes.py
Loading
Please register or sign in to comment