* Adjacency map - map from each op to its predecessor and successor ops
* More robust handling of Gemm nodes scope names (instead of
  increment_instance())
* More consistent handling of ops with the same scope name
* Handle pad + avg pool sequences generated by ONNX trace optimization
  (results in one less op in the graph, hence the changes in tests)
* Minor refactoring in predecessors() and successors() functions

Remove the multiple instances of code that generates
dummy input per dataset.

The weights_vol attribute reflects the size (volume) of an SG node’s
weights tensor.  The calculation of the weights volume was wrong.
This does not have any significant impact because this attribute is
not used.

This reverts commit a3f2ce2d.

The weights_vol attribute reflects the size (volume) of an SG node’s
weights tensor.  The calculation of the weights volume was wrong.
This does not have any significant impact because this attribute is
not used.
wq

Also added tests

This commit simplifies the SummaryGraph API,
by removing from the client to burden to handle 
the differences between models with/without 
DataParallel layers.

DataParallel layers in PyTorch change the fully-qualified
names (FQNs) of PyTorch modules.  A module's FQN
unambiguously identifies a module within a model, by 
encoding the path to the module from the root of the 
model.  For example, ```module.layer2.1.conv1``` and 
```module.layer2.0.conv1``` are FQNs of two different
modules named ```conv1``` in some module.  
Because a module's FQN reflects the module's hierarchy,
adding/removing a DataParallel node also changes its FQN.

Distiller uses FQNs to refer to modules and parameters 
(e.g. from YAML files), and non-functional changes to the 
model hierarchy, such as using DataParallel modules are
handled by converting FQNs using `
``utils.{de,}normalize_module_name()```.

Before this commit, the SummaryGraph API assumed that 
the API client will convert layers names using 
```utils.normalize_module_name()``` before invoking the API.
This led to needlessly verbose client code, which was also
error-prone and harder to read and maintain.
This commit fixes these short-comings by relaxing the API, 
and handling the FQNN naming differences internally.

The thinning implementation is simplified somewhat
by refactoring to the new APIs lenient requirements.

Added named_params_layers method to SummaryGraph
that yields a 3-tuple of: layer name, param name, and param.
When using the new method, summary graph communicates the
true layer name in respect to the model it was initiated with.

Dropout layers were not handled properly in SummaryGraph, and
caused the indexing of layer names to change.
The root cause is that in ONNX uses the same node name for
Dropout and Linear layers that are processed in sequence.
ONNX nodes can be identified by three components: the ONNX 
node name,  type, and instance.
In SummaryGraph we ignore the node type when naming a node.
Specifically in AlexNet, nodes the Dropout layers before a Linear
layer have the same node name and instance, and are only distinguished
by their type.  SummaryGraph, ignorant of the type, skipped the Dropout
layers and gave SG nodes the wrong name.  Thus 'classifier.0', which is
a Dropout node, became a Linear node.
The fix is not to ignore duplicate (node name, instance) pairs
by incrementing the instance.

Not backward compatible - re-installation is required

* Fixes for PyTorch==1.0.0
* Refactoring folder structure
* Update installation section in docs

Activation statistics can be leveraged to make pruning and quantization decisions, and so
We added support to collect these data.
- Two types of activation statistics are supported: summary statistics, and detailed records 
per activation.
Currently we support the following summaries: 
- Average activation sparsity, per layer
- Average L1-norm for each activation channel, per layer
- Average sparsity for each activation channel, per layer

For the detailed records we collect some statistics per activation and store it in a record.  
Using this collection method generates more detailed data, but consumes more time, so
Beware.

* You can collect activation data for the different training phases: training/validation/test.
* You can access the data directly from each module that you chose to collect stats for.  
* You can also create an Excel workbook with the stats.

To demonstrate use of activation collection we added a sample schedule which prunes 
weight filters by the activation APoZ according to:
"Network Trimming: A Data-Driven Neuron Pruning Approach towards 
Efficient Deep Architectures",
Hengyuan Hu, Rui Peng, Yu-Wing Tai, Chi-Keung Tang, ICLR 2016
https://arxiv.org/abs/1607.03250

We also refactored the AGP code (AutomatedGradualPruner) to support structure pruning,
and specifically we separated the AGP schedule from the filter pruning criterion.  We added
examples of ranking filter importance based on activation APoZ (ActivationAPoZRankedFilterPruner),
random (RandomRankedFilterPruner), filter gradients (GradientRankedFilterPruner), 
and filter L1-norm (L1RankedStructureParameterPruner)

Somehow 4 copies of the same test were pasted into this file: removed 3 instances.

This is a merge of the ADC branch and master.
ADC (using a DDPG RL agent to compress image classifiers) is still WiP and requires
An unreleased version of Coach (https://github.com/NervanaSystems/coach).

Small features in this commit:
-Added model_find_module() - find module object given its name
- Add channel ranking and pruning: pruning/ranked_structures_pruner.py
- Add a CIFAR10 VGG16 model: models/cifar10/vgg_cifar.py
- Thinning: change the level of some log messages – some of the messages were
moved to ‘debug’ level because they are not usually interesting.
- Add a function to print nicely formatted integers - distiller/utils.py
- Sensitivity analysis for channels-removal
- compress_classifier.py – handle keyboard interrupts
- compress_classifier.py – fix re-raise of exceptions, so they maintain call-stack

-Added tests:
-- test_summarygraph.py: test_simplenet() - Added a regression test to target a bug that occurs when taking the predecessor of the first node in a graph
-- test_ranking.py - test_ch_ranking, test_ranked_channel_pruning
-- test_model_summary.py - test_png_generation, test_summary (sparsity/ compute/model/modules)

- Bug fixes in this commit:
-- Thinning bug fix: handle zero-sized 'indices' tensor
During the thinning process, the 'indices' tensor can become zero-sized,
and will have an undefiend length. Therefore, we need to check for this
situation when assessing the number of elements in 'indices'
-- Language model: adjust main.py to new distiller.model_summary API

*connectivity_summary() does not use SummaryGraph correctly:
 Recently we changed the internal representation of SummaryGraph.ops, but
 connectivity_summary() and connectivity_summary_verbose() were not updated.
 Fixed that.

*Extend the API of create_png():
 Add to the signature of create_png() and create_pydot_graph() rankdir and
 External styles.  These are explained in the docstrings.

*Added documentation to the PNG drawing functions

*Added tests to catch trivial connectivity_summary() bugs

Fixed a bug in module name normalization, for modules with
a name ending in ".module" (e.g. "features.module" in the case of
VGG).
Made the tests more robust, and also refactored the common code
to distiller/utils.py

* Modify 'create_png' to use the correct data structures (dicts instead
  lists, etc.)
* Handle case where an op was called not from a module. This relates to:
  * ONNX->"User-Friendly" name conversion to account for cases where
  * Detection of existing op with same name
  In both cases use the ONNX op type in addition to the op name
* Return an "empty" shape instead of None when ONNX couldn't infer
  a parameter's shape
* Expose option of PNG summary with parameters to user

* Large update containing new thinning algorithm.

Thinning a model is the process of taking a dense network architecture with a parameter model that
has structure-sparsity (filters or channels) in the weights tensors of convolution layers, and making changes in the network architecture and parameters, in order to completely remove the structures.
The new architecture is smaller (condensed), with less channels and filters in some of the convolution layers.  Linear and BatchNormalization layers are also adjusted as required.

To perform thinning, we create a SummaryGraph (‘sgraph’) of our model.  We use the ‘sgraph’ to infer the
data-dependency between the modules in the PyTorch network.  This entire process is not trivial and will be
documented in a different place.

Large refactoring of SummaryGraph to support the new thinning requirement for traversing successors and predecessors.
- Operations (ops) are now stored in a dictionary, so that they can be accessed quickly by name.
- Refactor Operation construction code
- Added support for search a node’s predecessors and successors.  You can search for all predecessors/successors by depth, or by type.
- create_png now supports an option to display the parameter nodes

Updated schedules with new thinning syntax.

* Thinning: support iterative thinning of models

THere's a caveat with this commit: when using this code you will
need to train with SGD momentum=0.
The momentum update is dependent on the weights, and because we
dynamically change the weights shapes, we need to either make the
apporpriate changes in the Optimizer, or disable the momentum.
For now, we disable the momentum

* Thinning: move the application of FilterRemover to on_minibatch_begin

* Thinning: fix syntax error

* Word-level language model compression

Added an implementation of Baidu’s RNN pruning scheme:
Narang, Sharan & Diamos, Gregory & Sengupta, Shubho & Elsen, Erich. (2017).
    Exploring Sparsity in Recurrent Neural Networks.
    (https://arxiv.org/abs/1704.05119)

Added an example of word-level language model compression.
The language model is based on PyTorch’s example:
https://github.com/pytorch/examples/tree/master/word_language_model

Added an AGP pruning schedule and RNN pruning schedule to demonstrate
compression of the language model.

* thinning: remove dead code

* remove resnet18 filter pruning since the scheduler script is incomplete

* thinning: fix indentation error

* thinning: remove dead code

* thinning: updated resnet20-CIFAR filter-removsal reference checkpoints

* thinning: updated resnet20-CIFAR filter-removal reference schedules

These are for use with the new thinning scheudle algorithm