Skip to content
Snippets Groups Projects
  • Neta Zmora's avatar
    a0bf2a8f
    Important changes to pruning channels and filters (#93) · a0bf2a8f
    Neta Zmora authored
    This commit contains the main fix for issue #85.  It contains a couple of changes to the YAML structure pruning API, with examples.
    I urge you to read the documentation in the Wiki (https://github.com/NervanaSystems/distiller/wiki/Pruning-Filters-&-Channels).
    
    New syntax for defining Structured AGP.  I tried to make the syntax similar to fine-grained
    (i.e. element-wise) pruning.  All you need to do is add: ```group_type: Filters```.
    ```
      low_pruner:
        class: L1RankedStructureParameterPruner_AGP
        initial_sparsity : 0.10
        final_sparsity: 0.50
        group_type: Filters
        weights: [module.layer3.0.conv2.weight,
                  module.layer3.0.downsample.0.weight,
                  module.layer3.1.conv2.weight,
                  module.layer3.2.conv2.weight]
    ```
    
    If you want to define “leader-based” pruning dependencies, add ```group_dependency: Leader```:
    ```
      low_pruner:
        class: L1RankedStructureParameterPruner_AGP
        initial_sparsity : 0.10
        final_sparsity: 0.50
        group_type: Filters
        group_dependency: Leader
        weights: [module.layer3.0.conv2.weight,
                  module.layer3.0.downsample.0.weight,
                  module.layer3.1.conv2.weight,
                  module.layer3.2.conv2.weight]
    ```
    
    Retired the old ```reg_regims``` API for describing one-shot structured-pruning.
    
    The new YAML API is very similar to AGP structured-pruning, which is much better
    than before.
    The new API also allows us to describe data-dependencies when doing one-shot
    structure pruning, just like AGP structured-pruning.
    
    This commit also includes further code refactoring.
    
    Old API:
    ```
      filter_pruner:
         class: 'L1RankedStructureParameterPruner'
         reg_regims:
           'module.layer1.0.conv1.weight': [0.6, '3D']
           'module.layer1.1.conv1.weight': [0.6, '3D']
    ```
    
    New API:
    ```
     filter_pruner:
        class: 'L1RankedStructureParameterPruner'
        group_type: Filters
        desired_sparsity: 0.6
        weights: [
          module.layer1.0.conv1.weight,
          module.layer1.1.conv1.weight]
    ```
    
    thresholding.py – separate the generation of the binary_map from the pruning_mask so that we
    can cache the binary map and share it between several modules.
    
    pruning/automated_gradual_pruner.py – major refactoring to supported “leader-based”
    sub-graph pruning dependencies.  The concept is explained in issue #85
    
    
    agp-pruning/resnet20_filters.schedule_agp.yaml
    agp-pruning/resnet20_filters.schedule_agp_2.yaml
    agp-pruning/resnet20_filters.schedule_agp_3.yaml
    network_trimming/resnet56_cifar_activation_apoz.yaml
    network_trimming/resnet56_cifar_activation_apoz_v2.yaml
    Important changes to pruning channels and filters (#93)
    Neta Zmora authored
    This commit contains the main fix for issue #85.  It contains a couple of changes to the YAML structure pruning API, with examples.
    I urge you to read the documentation in the Wiki (https://github.com/NervanaSystems/distiller/wiki/Pruning-Filters-&-Channels).
    
    New syntax for defining Structured AGP.  I tried to make the syntax similar to fine-grained
    (i.e. element-wise) pruning.  All you need to do is add: ```group_type: Filters```.
    ```
      low_pruner:
        class: L1RankedStructureParameterPruner_AGP
        initial_sparsity : 0.10
        final_sparsity: 0.50
        group_type: Filters
        weights: [module.layer3.0.conv2.weight,
                  module.layer3.0.downsample.0.weight,
                  module.layer3.1.conv2.weight,
                  module.layer3.2.conv2.weight]
    ```
    
    If you want to define “leader-based” pruning dependencies, add ```group_dependency: Leader```:
    ```
      low_pruner:
        class: L1RankedStructureParameterPruner_AGP
        initial_sparsity : 0.10
        final_sparsity: 0.50
        group_type: Filters
        group_dependency: Leader
        weights: [module.layer3.0.conv2.weight,
                  module.layer3.0.downsample.0.weight,
                  module.layer3.1.conv2.weight,
                  module.layer3.2.conv2.weight]
    ```
    
    Retired the old ```reg_regims``` API for describing one-shot structured-pruning.
    
    The new YAML API is very similar to AGP structured-pruning, which is much better
    than before.
    The new API also allows us to describe data-dependencies when doing one-shot
    structure pruning, just like AGP structured-pruning.
    
    This commit also includes further code refactoring.
    
    Old API:
    ```
      filter_pruner:
         class: 'L1RankedStructureParameterPruner'
         reg_regims:
           'module.layer1.0.conv1.weight': [0.6, '3D']
           'module.layer1.1.conv1.weight': [0.6, '3D']
    ```
    
    New API:
    ```
     filter_pruner:
        class: 'L1RankedStructureParameterPruner'
        group_type: Filters
        desired_sparsity: 0.6
        weights: [
          module.layer1.0.conv1.weight,
          module.layer1.1.conv1.weight]
    ```
    
    thresholding.py – separate the generation of the binary_map from the pruning_mask so that we
    can cache the binary map and share it between several modules.
    
    pruning/automated_gradual_pruner.py – major refactoring to supported “leader-based”
    sub-graph pruning dependencies.  The concept is explained in issue #85
    
    
    agp-pruning/resnet20_filters.schedule_agp.yaml
    agp-pruning/resnet20_filters.schedule_agp_2.yaml
    agp-pruning/resnet20_filters.schedule_agp_3.yaml
    network_trimming/resnet56_cifar_activation_apoz.yaml
    network_trimming/resnet56_cifar_activation_apoz_v2.yaml