Force loading on the CPU which always has more memory than a
single GPU.  This is useful for models that cannot be loaded onto
a single GPU.

As documented in issue #395, some of the command-line examples in the
AMC notebooks are incorrect.
Also, fix some bugs that were introduced with the refactoring of the
low-level pruning API

(since stats collection is now a 2-phase process)

add citation

* Greedy search script for post-training quantization settings
  * Iterates over each layer in the model in order. For each layer,
    checks a user-defined set of quantization settings and chooses
    the best one based on validation accuracy
  * Provided sample that searches for best activations-clipping
    mode per layer, on image classification models

* Proper handling of mixed-quantization settings in post-train quant:
  * By default, the quantization settings for each layer apply only
    to output quantization
  * Propagate quantization settings for activations tensors through
    the model during execution
  * For non-quantized inputs to layers that require quantized inputs,
    fall-back to quantizing according to the settings used for the
    output
  * In addition, provide mechanism to override inputs quantization
    settings via the YAML configuration file
  * By default all modules are quantized now. For module types that
    don't have a dedicated quantized implementation, "fake"
    quantization is performed

* Misc. Changes
  * Fuse ReLU/ReLU6 to predecessor during post-training quantization
  * Fixes to ACIQ clipping in the half-range case

Co-authored-by: Lev Zlotnik <lev.zlotnik@intel.com>
Co-authored-by: Guy Jacob <guy.jacob@intel.com>

As noted in issue #382, logging when a parameter does not have
a mask is unnecessary and may confuse users.  Therefore, it is
removed.

`app_cfg` logs the basic execution environment state, and is deemed important
in most circumstances.

Some refactoring of the low-level pruning API

Added distiller/norms.py - for calculating norms of various sub-tensors.

ranked_structures_pruner.py:
-Removed l1_magnitude, l2_magnitude. Use instead distiller.norms.l1_norm
-Lots of refactoring
-replaced LpRankedStructureParameterPruner.ch_binary_map_to_mask with
distiller.thresholding.expand_binary_map
-FMReconstructionChannelPruner.rank_and_prune_channels used L2-norm
by default and now uses L1-norm (i.e.magnitude_fn=l2_magnitude was
replaced with magnitude_fn=distiller.norms.l1_norm)

thresholding.py:
-Delegated lots of the work to the new norms.py.
-Removed support for 4D (entire convolution layers) since that has not been
maintained for a longtime. This may break some old scripts that remove entire
layers.
-added expand_binary_map() explicitly so others can use it. Might need to
move to a different file
-removed threshold_policy()

utils.py:
-use distiller.norms.xxx for sparsity stats

Hot-fix for issue that arises with FileWriter class on TF v2.
Allows only Tensorflow v1.X

* Create float copy such that the actual tensor being learned stays
  the same
* This way the optimizer doesn't have to be re-created, just need to
  add parameter groups if algo requires it (e.g. PACT)
* This also means we don't care about pre-existing parameter groups,
  as opposed to the previous implementation which ASSUMED a single
  existing group

Move these files to their true location, instead of using soft-links.

Also added a short README file to distiller/examples/baseline_networks
directory.

The PR was based on an older code-base, which contained older code
and which was accidentally merged with the PR

* And removed unnecessary argument from execution env logging function

* Return both min and max clip value in the symmetric case
* Correct delta from mean in asymmetric + half_range case

Add a jupyter notebook showing how to register a user's (external) image-classification model.
Contains fixes to the previous models extension mechanism, and relaxation of the `args' requirements in apputils/image_classifier.py.

apputils/image_classifier.py –
*when self.logdir is None:
-use NullLogger
-skip save_checkpoint

*return training log from run_training_loop()
*don’t log if script_dir or output_dir are not set.
*Fix params_nnz_cnt in update_training_scores_history()

data_loggers/logger.py – add NullLogger which does not log

../../../distiller/apputils/image_classifier.py – remove unused
`--extras` command-line argument

Odds and ends commit

A bundle of very small, and mostly non-functional, changes to the code.
Mostly they are unrelated to each other
../../../distiller/apputils/checkpoint.py – add info to exception

../../../distiller/apputils/image_classifier.py – remove unused
`--extras` command-line argument

../../../distiller/thinning.py – code refactoring (non-functional)
except for adding a new public API: contract_model()

../../classifier_compression/compress_classifier.py – use
contract_model() when using `--thinnify`

../../lottery_ticket/README.md – remove illegal characters in
the text

ACIQ clipping method, as described in:

Post training 4-bit quantization of convolutional networks for rapid-deployment
(Ron Banner , Yury Nahshan, Daniel Soudry)
(NeurIPS 2019)

https://arxiv.org/abs/1810.05723



Co-authored-by: Yury Nahshan <yury.nahshan@intel.com>
Co-authored-by: Lev Zlotnik <lev.zlotnik@intel.com>

Added explicit thank you mention for Cadene

This patch moves the check on remaining training epochs into the training loop, as it is irrelevant for non-training invocation of ClassifierCompressor.
Thus, allowing resumed checkpoints be evaluated regardless of the epochs limitation.

This patch moves the check on remaining training epochs into the training loop, as it is irrelevant for non-training invocation of ClassifierCompressor.
Thus, allowing resumed checkpoints be evaluated regardless of the epochs limitation.

* The is_simulated_quant_weight_shifted flag was a Python bool,
  modifying of which during forward() isn't supported when 
  DataParallel is used, as per
  https://pytorch.org/docs/stable/nn.html#dataparallel-layers-multi-gpu-distributed
* Instead, registered it as an integer tensor buffer

Integrate the code for the DDPG agent from:
https://github.com/mit-han-lab/amc-release

The instructions for cloning HAN's code and then making changes
to fit Distiller were too complicated, so we added the integrated
files to distiller/examples/auto_compression/amc/rl_lib/hanlab