Some refactoring of the low-level pruning API

Added distiller/norms.py - for calculating norms of various sub-tensors.

ranked_structures_pruner.py:
-Removed l1_magnitude, l2_magnitude. Use instead distiller.norms.l1_norm
-Lots of refactoring
-replaced LpRankedStructureParameterPruner.ch_binary_map_to_mask with
distiller.thresholding.expand_binary_map
-FMReconstructionChannelPruner.rank_and_prune_channels used L2-norm
by default and now uses L1-norm (i.e.magnitude_fn=l2_magnitude was
replaced with magnitude_fn=distiller.norms.l1_norm)

thresholding.py:
-Delegated lots of the work to the new norms.py.
-Removed support for 4D (entire convolution layers) since that has not been
maintained for a longtime. This may break some old scripts that remove entire
layers.
-added expand_binary_map() explicitly so others can use it. Might need to
move to a different file
-removed threshold_policy()

utils.py:
-use distiller.norms.xxx for sparsity stats

Move these files to their true location, instead of using soft-links.

Also added a short README file to distiller/examples/baseline_networks
directory.

* And removed unnecessary argument from execution env logging function

A bundle of very small, and mostly non-functional, changes to the code.
Mostly they are unrelated to each other
../../../distiller/apputils/checkpoint.py – add info to exception

../../../distiller/apputils/image_classifier.py – remove unused
`--extras` command-line argument

../../../distiller/thinning.py – code refactoring (non-functional)
except for adding a new public API: contract_model()

../../classifier_compression/compress_classifier.py – use
contract_model() when using `--thinnify`

../../lottery_ticket/README.md – remove illegal characters in
the text

ACIQ clipping method, as described in:

Post training 4-bit quantization of convolutional networks for rapid-deployment
(Ron Banner , Yury Nahshan, Daniel Soudry)
(NeurIPS 2019)

https://arxiv.org/abs/1810.05723



Co-authored-by: Yury Nahshan <yury.nahshan@intel.com>
Co-authored-by: Lev Zlotnik <lev.zlotnik@intel.com>

Integrate the code for the DDPG agent from:
https://github.com/mit-han-lab/amc-release

The instructions for cloning HAN's code and then making changes
to fit Distiller were too complicated, so we added the integrated
files to distiller/examples/auto_compression/amc/rl_lib/hanlab

Mainly: moved NetworkWrapper to a separate file.

FMReconstructionChannelPruner: add support for nn.Linear layers
utils.py: add non_zero_channels()
thinning: support removing channels from FC layers preceding Conv layers
test_pruning.py: add test_row_pruning()
scheduler: init from a dictionary of Maskers
coach_if.py – fix imports of Clipped-PPO and TD3

This command command-line argument allows us to save the
randomly-initialized model before training (useful for lottery-ticket
method).
This commit was accidentally left out of the Lottery-Ticket Hypothesis
commit from Aug 26.

Added support for saving the randomly initialized network before
starting training; and added an implmentation showing how to extract
a (winning) lottery ticket from the prestine network, and the
pruned network.

When using flag `-s` which prints the compression scheduler
pruning mask keys, we also print a table with the fine-grain
sparsity of each mask.

*An implementation of AMC (the previous implementation
 code has moved to a new location under 
/distiller/examples/auto_compression/amc.  AMC is aligned
with the ‘master’ branch of Coach.
*compress_classifier.py is refactored.  The base code moved
to /distiller/apputils/image_classifier.py.  Further refactoring
will follow.
We want to provide a simple and small API to the basic features of
a classifier-compression application.
This will help applications that want to use the make features of a
classifier-compression application, without the standard training
regiment.
AMC is one example of a stand-alone application that needs to leverage
the capabilities of a classifier-compression application, but is currently
coupled to `compress_classifier.py`.
`multi-finetune.py` is another example.
* ranked_structures_pruner.py:
** Added support for grouping channels/filters
Sometimes we want to prune a group of structures: e.g. groups of
8-channels.  This feature does not force the groups to be adjacent,
so it is more like a set of structures.  E.g. in the case of pruning
channels from a 64-channels convolution, grouped by 8 channels, we 
will prune exactly one of 0/8/16/24/32/40/48/56 channels.  I.e. 
always a multiple of 8-channels, excluding the set of all 64 channels.
** Added FMReconstructionChannelPruner – this is channel
pruning using L1-magnitude to rank and select channels to
remove, and feature-map reconstruction to improve the
resilience to the pruning.
* Added a script to run multiple instances of an 
experiment, in different processes:
 examples/classifier_compression/multi-run.py
* Set the seed value even when not specified by the command-line
arguments, so that we can try and recreate the session.
* Added pruning ranking noise -
Ranking noise introduces Gaussian noise when ranking channels/filters
using Lp-norm.  The noise is introduced using the epsilon-greedy
methodology, where ranking using exact Lp-norm is considered greedy.
* Added configurable rounding of pruning level: choose whether to 
Round up/down when rounding the number of structures to prune 
(rounding is always to an integer).  

* Pass the 'sigmoid' flag in NeuMF.forward as a bool tensor instead of
  a simple boolean. Required to make the model traceable (it?d be better
  to not have it an argument of forward at all, but keeping changes to
  a minimum)
* Call prepare_model with dummy_input

The PyTorch trace mechanism doesn't "see" torch.nn.ModuleList modules
(since they don't have a forward function). As a result, the mapping
from module names at the Python model definition level to the
scope-names at the trace level is not 1:1. This makes it impossible for
us to map back from SummaryGraph ops to their respective nn.Modules,
which is required for flows like BatchNorm folding and stats fusion in
post-training quantization.

In #313 we handled this issue specifically in DistillerLSTM, but it
makes much more sense to have a generic and automatic solution for this
issue, which doesn't require the user to modify the model. This is such
a solution.
    
* Implemented DistillerModuleList, a replacement for nn.ModuleList
  which results in full and unique scope-names
* See documentation for this class in summary_graph.py for extensive
  details on the issue and solution
* When generating a SummaryGraph, the model is scanned and all instances
  of torch.nn.ModuleList are replaced with DistillerModulelist
* Add tests for new functionality
* Partially revert changes made to DistillerLSTM in commit 43548deb:
  Keep the refactored _create_cells_list function, but have it create
  a standard torch.nn.ModuleList (since we're the ModuleList issue
  automatically now, and no need to confuse users with ad-hoc list 
  implementations

* "Net-aware quantization" - using the term coined in
  https://arxiv.org/abs/1811.09886. (section 3.2.2).
  Refers to considering sequences of modules when quantizing. This 
  isn't exactly layer fusion - we modify activation stats prior to
  setting quantization parameters, to make sure that when a module
  is followed by certain activation functions, only the relevant
  ranges are quantized. We do this for:
    * ReLU - Clip all negative values
    * Tanh / Sigmoid - Clip according to the (approximated) saturation
      values for these functions. We use [-4, 4] for tanh and [-6, 6]
      for sigmoid.

* Perform batch-norm folding before post-training quantization.
  Batch-norm parameters are folded into the parameters of the previous
  layer and the BN layer is replaced with an identity module.

* Both BN folding and "net-aware" are now automatically executed
  in PostTrainLinearQuantizer (details of this change below)

* BN folding enabled by new generic mechanism to "fuse" module
  sequences (at the Python API level)
    * First module in sequence is replaced/modified by a user-provided
      function, rest of moudles replaced with nn.Identity

* Quantizer changes:
  * Optionally create adjacency map during prepare_model
  * Subclasses may enforce adjacency map creation
  * Refatcoring: Replace _prepare_model_impl with pre and post
    override-able "callbacks", so core functionality is always executed

* PostTrainLinearQuantizer Changes:
  * Enforce creation of adjacency map. This means users must now pass a
    dummy input to PostTrainLinearQuantizer.prepare_model
  * Before module replacement - Apply BN folding and stats updates according
    to net-aware quantization

* Updated the language model quantization tutorial to reflect the new
  functionality

* Updated the image classification post-train quantization samples
  (command line and YAML)

* Other changes:
  * Distller LSTM implementation:
    Replace the ModuleList for cells with a plain list. The PyTorch trace
    mechanism doesn't "see" ModuleList objects, it only sees the 
    contained modules. This means that the "scopeName" of these modules
    isn't complete, which makes it impossible to match op names in 
    SummaryGraph to modules in the Python model.
  * ActivationStatsCollector: Ignore nn.Identity modules

* PyTorch 1.1.0 now required
  - Moved other dependencies to up-to-date versions as well
* Adapt LR scheduler to PyTorch 1.1 API changes:
  - Change lr_scheduler.step() calls to succeed validate calls,
    during training
  - Pass to lr_scheduler.step() caller both loss and top1
    (Resolves issue #240)
* Adapt thinning for PyTorch 1.1 semantic changes
  - **KNOWN ISSUE**: When a thinning recipe is applied, in certain
    cases PyTorch displays this warning:
    "UserWarning: non-inplace resize is deprecated".
    To be fixed later
* SummaryGraph: Workaround for new scope name issue from PyTorch 1.1.0
* Adapt to updated PyTest version:
  - Stop using deprecated 'message' parameter of pytest.raises(),
    use pytest.fail() instead
  - Make sure only a single test case per pytest.raises context
* Move PyTorch version check to root __init__.py 
  - This means the version each checked when Distiller is first
    imported. A RuntimeError is raised if the version is wrong.
* Updates to parameter_histograms notebook:
  - Replace deprecated normed argument with density
  - Add sparsity rate to plot title
  - Load model in CPU

* In PostTrainLinearQuantizer - moved 'clip_acts' and 'clip_n_stds'
  to overrides, removed 'no_clip_layers' parameter from __init__
* The 'no_clip_layers' command line argument REMAINS, handled in 
  PostTrainLinearQuantizer.from_args()
* Removed old code from comments, fixed warnings in 
  test_post_train_quant.py
* Updated tests
* Update post-train quant sample YAML

Also added a simple network model for MNIST, under
distiller/models/mnist.

Added set_seed() to Distiller and added support for seeding the PRNG when setting --deterministic mode (prior to this change, the seed is always set to zero when running in deterministic mode.
The PRNGs of Pytorch (CPU & Cuda devices), numpy and Python are set.
Added support for ```--seed``` to classifier_compression.py.