Skip to content
Snippets Groups Projects
  • Neta Zmora's avatar
    42650340
    Thinning (#6) · 42650340
    Neta Zmora authored
    * Large update containing new thinning algorithm.
    
    Thinning a model is the process of taking a dense network architecture with a parameter model that
    has structure-sparsity (filters or channels) in the weights tensors of convolution layers, and making changes in the network architecture and parameters, in order to completely remove the structures.
    The new architecture is smaller (condensed), with less channels and filters in some of the convolution layers.  Linear and BatchNormalization layers are also adjusted as required.
    
    To perform thinning, we create a SummaryGraph (‘sgraph’) of our model.  We use the ‘sgraph’ to infer the
    data-dependency between the modules in the PyTorch network.  This entire process is not trivial and will be
    documented in a different place.
    
    Large refactoring of SummaryGraph to support the new thinning requirement for traversing successors and predecessors.
    - Operations (ops) are now stored in a dictionary, so that they can be accessed quickly by name.
    - Refactor Operation construction code
    - Added support for search a node’s predecessors and successors.  You can search for all predecessors/successors by depth, or by type.
    - create_png now supports an option to display the parameter nodes
    
    Updated schedules with new thinning syntax.
    
    * Thinning: support iterative thinning of models
    
    THere's a caveat with this commit: when using this code you will
    need to train with SGD momentum=0.
    The momentum update is dependent on the weights, and because we
    dynamically change the weights shapes, we need to either make the
    apporpriate changes in the Optimizer, or disable the momentum.
    For now, we disable the momentum
    
    * Thinning: move the application of FilterRemover to on_minibatch_begin
    
    * Thinning: fix syntax error
    
    * Word-level language model compression
    
    Added an implementation of Baidu’s RNN pruning scheme:
    Narang, Sharan & Diamos, Gregory & Sengupta, Shubho & Elsen, Erich. (2017).
        Exploring Sparsity in Recurrent Neural Networks.
        (https://arxiv.org/abs/1704.05119)
    
    Added an example of word-level language model compression.
    The language model is based on PyTorch’s example:
    https://github.com/pytorch/examples/tree/master/word_language_model
    
    Added an AGP pruning schedule and RNN pruning schedule to demonstrate
    compression of the language model.
    
    * thinning: remove dead code
    
    * remove resnet18 filter pruning since the scheduler script is incomplete
    
    * thinning: fix indentation error
    
    * thinning: remove dead code
    
    * thinning: updated resnet20-CIFAR filter-removsal reference checkpoints
    
    * thinning: updated resnet20-CIFAR filter-removal reference schedules
    
    These are for use with the new thinning scheudle algorithm
    42650340
    History
    Thinning (#6)
    Neta Zmora authored
    * Large update containing new thinning algorithm.
    
    Thinning a model is the process of taking a dense network architecture with a parameter model that
    has structure-sparsity (filters or channels) in the weights tensors of convolution layers, and making changes in the network architecture and parameters, in order to completely remove the structures.
    The new architecture is smaller (condensed), with less channels and filters in some of the convolution layers.  Linear and BatchNormalization layers are also adjusted as required.
    
    To perform thinning, we create a SummaryGraph (‘sgraph’) of our model.  We use the ‘sgraph’ to infer the
    data-dependency between the modules in the PyTorch network.  This entire process is not trivial and will be
    documented in a different place.
    
    Large refactoring of SummaryGraph to support the new thinning requirement for traversing successors and predecessors.
    - Operations (ops) are now stored in a dictionary, so that they can be accessed quickly by name.
    - Refactor Operation construction code
    - Added support for search a node’s predecessors and successors.  You can search for all predecessors/successors by depth, or by type.
    - create_png now supports an option to display the parameter nodes
    
    Updated schedules with new thinning syntax.
    
    * Thinning: support iterative thinning of models
    
    THere's a caveat with this commit: when using this code you will
    need to train with SGD momentum=0.
    The momentum update is dependent on the weights, and because we
    dynamically change the weights shapes, we need to either make the
    apporpriate changes in the Optimizer, or disable the momentum.
    For now, we disable the momentum
    
    * Thinning: move the application of FilterRemover to on_minibatch_begin
    
    * Thinning: fix syntax error
    
    * Word-level language model compression
    
    Added an implementation of Baidu’s RNN pruning scheme:
    Narang, Sharan & Diamos, Gregory & Sengupta, Shubho & Elsen, Erich. (2017).
        Exploring Sparsity in Recurrent Neural Networks.
        (https://arxiv.org/abs/1704.05119)
    
    Added an example of word-level language model compression.
    The language model is based on PyTorch’s example:
    https://github.com/pytorch/examples/tree/master/word_language_model
    
    Added an AGP pruning schedule and RNN pruning schedule to demonstrate
    compression of the language model.
    
    * thinning: remove dead code
    
    * remove resnet18 filter pruning since the scheduler script is incomplete
    
    * thinning: fix indentation error
    
    * thinning: remove dead code
    
    * thinning: updated resnet20-CIFAR filter-removsal reference checkpoints
    
    * thinning: updated resnet20-CIFAR filter-removal reference schedules
    
    These are for use with the new thinning scheudle algorithm