Release 0.3 broke the expots to PNG and ONNX and this is the fix.

Not backward compatible - re-installation is required

* Fixes for PyTorch==1.0.0
* Refactoring folder structure
* Update installation section in docs

A small change to support ranking weight filters by the mean mean-value
of the feature-map channels.
Mean mean-value refers to computing the average value (across many
input images) of the mean-value of each channel.

Modified log_execution_env_state() to store
configuration file in the output directory,
under 'configs' sub-directory it creates.

At this time, the only configuration file is
passed via args.compress

To use automated compression you need to install several optional packages
which are not required for other use-cases.
This fix hides the import requirements for users who do not want to install
the extra packages.

Merging the 'amc' branch with 'master'.
This updates the automated compression code in 'master', and adds a greedy filter-pruning algorithm.

Summary of changes:
(1) Post-train quantization based on pre-collected statistics
(2) Quantized concat, element-wise addition / multiplication and embeddings
(3) Move post-train quantization command line args out of sample code
(4) Configure post-train quantization from YAML for more fine-grained control

(See PR #136 for more detailed changes descriptions)

* For CIFAR-10 / ImageNet only
* Refactor data_loaders.py, reduce code duplication
* Implemented custom sampler
* Integrated in image classification sample
* Since we now shuffle the test set, had to update expected results
  in 2 full_flow_tests that do evaluation

* Support for multi-phase activations logging

Enable logging activation both durning training and validation at
the same session.

* Refactoring: Move parser to its own file

* Parser is moved from compress_classifier into its own file.
* Torch version check is moved to precede main() call.
* Move main definition to the top of the file.
* Modify parser choices to case-insensitive

Fix a mismatch between the location of the model and the
computation.

In compress_classifier.py we added a new application argument: --cpu
which you can use to force compute (training/inference) to run on the CPU 
when you invoke compress_classifier.py on a machine which has Nvidia GPUs.

If your machine lacks Nvidia GPUs, then the compute will now run on the CPU
(and you do not need the new flag).

Caveat: we did not fully test the CPU support for the code in the Jupyter 
notebooks.  If you find a bug, we apologize and appreciate your feedback.

If compression_scheduler==None, then we need to set the value of
losses[OVERALL_LOSS_KEY] (so it is the same as losses[OBJECTIVE_LOSS_KEY]).
This was overlooked.

Added notebook for visualizing the discovery of compressed networks.
Added one-epoch fine-tuning at the end of every episode, which is
required for very sensitive models like Plain20.

Revert back to Pytorch 0.4.0. 
Also fixed some numpy calls (for statistics) that needed to be moved back to CPU.

Update the examples of earlyexit arguments which were not consistent with descriptions

* Asymmetric post-training quantization (only symmetric supported so until now)
* Quantization aware training for range-based (min-max) symmetric and asymmetric quantization
* Per-channel quantization support in both training and post-training
* Added tests and examples
* Updated documentation

This commit contains the main fix for issue #85.  It contains a couple of changes to the YAML structure pruning API, with examples.
I urge you to read the documentation in the Wiki (https://github.com/NervanaSystems/distiller/wiki/Pruning-Filters-&-Channels).

New syntax for defining Structured AGP.  I tried to make the syntax similar to fine-grained
(i.e. element-wise) pruning.  All you need to do is add: ```group_type: Filters```.
```
  low_pruner:
    class: L1RankedStructureParameterPruner_AGP
    initial_sparsity : 0.10
    final_sparsity: 0.50
    group_type: Filters
    weights: [module.layer3.0.conv2.weight,
              module.layer3.0.downsample.0.weight,
              module.layer3.1.conv2.weight,
              module.layer3.2.conv2.weight]
```

If you want to define “leader-based” pruning dependencies, add ```group_dependency: Leader```:
```
  low_pruner:
    class: L1RankedStructureParameterPruner_AGP
    initial_sparsity : 0.10
    final_sparsity: 0.50
    group_type: Filters
    group_dependency: Leader
    weights: [module.layer3.0.conv2.weight,
              module.layer3.0.downsample.0.weight,
              module.layer3.1.conv2.weight,
              module.layer3.2.conv2.weight]
```

Retired the old ```reg_regims``` API for describing one-shot structured-pruning.

The new YAML API is very similar to AGP structured-pruning, which is much better
than before.
The new API also allows us to describe data-dependencies when doing one-shot
structure pruning, just like AGP structured-pruning.

This commit also includes further code refactoring.

Old API:
```
  filter_pruner:
     class: 'L1RankedStructureParameterPruner'
     reg_regims:
       'module.layer1.0.conv1.weight': [0.6, '3D']
       'module.layer1.1.conv1.weight': [0.6, '3D']
```

New API:
```
 filter_pruner:
    class: 'L1RankedStructureParameterPruner'
    group_type: Filters
    desired_sparsity: 0.6
    weights: [
      module.layer1.0.conv1.weight,
      module.layer1.1.conv1.weight]
```

thresholding.py – separate the generation of the binary_map from the pruning_mask so that we
can cache the binary map and share it between several modules.

pruning/automated_gradual_pruner.py – major refactoring to supported “leader-based”
sub-graph pruning dependencies.  The concept is explained in issue #85


agp-pruning/resnet20_filters.schedule_agp.yaml
agp-pruning/resnet20_filters.schedule_agp_2.yaml
agp-pruning/resnet20_filters.schedule_agp_3.yaml
network_trimming/resnet56_cifar_activation_apoz.yaml
network_trimming/resnet56_cifar_activation_apoz_v2.yaml

* Fix issue #79

Change the default values so that the following scheduler meta-data keys
are always defined: 'starting_epoch', 'ending_epoch', 'frequency'

* compress_classifier.py: add a new argument

Allow the specification, from the command line arguments,  of the range of
pruning levels scanned when doing sensitivity analysis

* Add regression test for issue #79

Trying to simplify the code.

* Bug fix: value of best_top1 stored in the checkpoint may be wrong

If you invoke compress_clasifier.py with --num-best-scores=n
with n>1, then the value of best_top1 stored in checkpoints is wrong.

When we resume from a checkpoint, we usually want to continue using the checkpoint’s
masks.  I say “usually” because I can see a situation where we want to prune a model
and checkpoint it, and then resume with the intention of fine-tuning w/o keeping the
masks.  This is what’s done in Song Han’s Dense-Sparse-Dense (DSD) training
(https://arxiv.org/abs/1607.04381).  But I didn’t want to add another argument to
```compress_classifier.py``` for the time being – so we ignore DSD.

There are two possible situations when we resume a checkpoint that has a serialized
```CompressionScheduler``` with pruning masks:
1. We are planning on using a new ```CompressionScheduler``` that is defined in a
schedule YAML file.  In this case, we want to copy the masks from the serialized
```CompressionScheduler``` to the new ```CompressionScheduler``` that we are
constructing from the YAML file.  This is one fix.
2. We are resuming a checkpoint, but without using a YAML schedule file.
In this case we want to use the ```CompressionScheduler``` that we loaded from the
checkpoint file.  All this ```CompressionScheduler``` does is keep applying the masks
as we train, so that we don’t lose them.  This is the second fix.

For DSD, we would need a new flag that would override using the ```CompressionScheduler```
that we load from the checkpoint.

* Updated stats computation - fixes issues with validation stats

* Clarification of output (docs)

* Update

* Moved validation stats to separate function

By default, when we create a model we  wrap it with DataParallel to benefit
from data-parallelism across GPUs (mainly for convolution layers).

But sometimes we don't want the sample application to do this: for
example when we receive a model that was trained serially.
This commit adds a new argument to the application to prevent
the use of DataParallel.

* Fixed validation stats and added new summary stats

* Trimmed some comments.

* Improved figure for documentation

* Minor updates

Added an implementation of:

Dynamic Network Surgery for Efficient DNNs, Yiwen Guo, Anbang Yao, Yurong Chen.
NIPS 2016, https://arxiv.org/abs/1608.04493.

- Added SplicingPruner: A pruner that both prunes and splices connections.
- Included an example schedule on ResNet20 CIFAR.
- New features for compress_classifier.py:
   1. Added the "--masks-sparsity" which, when enabled, logs the sparsity
      of the weight masks during training.
  2. Added a new command-line argument to report the top N
      best accuracy scores, instead of just the highest score.
      This is sometimes useful when pruning a pre-trained model,
      that has the best Top1 accuracy in the first few pruning epochs.
- New features for PruningPolicy:
   1. The pruning policy can use two copies of the weights: one is used during
       the forward-pass, the other during the backward pass.
       This is controlled by the “mask_on_forward_only” argument.
   2. If we enable “mask_on_forward_only”, we probably want to permanently apply
       the mask at some point (usually once the pruning phase is done).
       This is controlled by the “keep_mask” argument.
   3. We introduce a first implementation of scheduling at the training-iteration
       granularity (i.e. at the mini-batch granularity). Until now we could schedule
       pruning at the epoch-granularity. This is controlled by the “mini_batch_pruning_frequency”
       (disable by setting to zero).

   Some of the abstractions may have leaked from PruningPolicy to CompressionScheduler.
   Need to reexamine this in the future.

* Added command line arguments for this and other post-training
  quantization settings in image classification sample.

Activation statistics can be leveraged to make pruning and quantization decisions, and so
We added support to collect these data.
- Two types of activation statistics are supported: summary statistics, and detailed records 
per activation.
Currently we support the following summaries: 
- Average activation sparsity, per layer
- Average L1-norm for each activation channel, per layer
- Average sparsity for each activation channel, per layer

For the detailed records we collect some statistics per activation and store it in a record.  
Using this collection method generates more detailed data, but consumes more time, so
Beware.

* You can collect activation data for the different training phases: training/validation/test.
* You can access the data directly from each module that you chose to collect stats for.  
* You can also create an Excel workbook with the stats.

To demonstrate use of activation collection we added a sample schedule which prunes 
weight filters by the activation APoZ according to:
"Network Trimming: A Data-Driven Neuron Pruning Approach towards 
Efficient Deep Architectures",
Hengyuan Hu, Rui Peng, Yu-Wing Tai, Chi-Keung Tang, ICLR 2016
https://arxiv.org/abs/1607.03250

We also refactored the AGP code (AutomatedGradualPruner) to support structure pruning,
and specifically we separated the AGP schedule from the filter pruning criterion.  We added
examples of ranking filter importance based on activation APoZ (ActivationAPoZRankedFilterPruner),
random (RandomRankedFilterPruner), filter gradients (GradientRankedFilterPruner), 
and filter L1-norm (L1RankedStructureParameterPruner)

Temporary fix for dependency on distiller class hierarchy when
serializing a model that contains a thinning recipe.

* Clean up PyTorch 0.3 compatibility code
We don't need this anymore and PyTorch 1.0 is just around the corner.

* explicitly place the inputs tensor on the GPU(s)

* A temporary fix for issue 36

The thinning code assumes that the sgraph it is using
is not data-parallel, because it (currently) accesses the
layer-name keys using a "normalized" name ("module." is removed).

The bug is that in thinning.py#L73 we create a data_parallel=True
model; and then give it to sgraph.
But in other places thinning code uses "normalized" keys.  For
example in thinning.py#L264.

The temporary fix configures data_parallel=False in thinning.py#L73.

A long term solution should have SummaryGraph know how to handle
both parallel and not-parallel models.  This can be done by having
SummaryGraph convert layer-names it receives in the API to
data_parallel=False using normalize_layer_name.  When returning
results, use the de-normalized format.

* Fix the documentation error from issue 36
* Move some logs to debug and show in logging.conf how to enable DEBUG logs.