Fixed two long-standing bugs in knowledge distillation:
 * Distillation loss needs to be scaled by T^2 (#122)
 * Use tensor.clone instead of new_tensor when caching student logits (#234)
Updated example results and uploaded the script to generate them

See issue #444

* Merge pytorch 1.3 commits

This PR is a fix for issue #422.

1. ImageNet models usually use input size [batch, 3, 224, 224], but all Inception models require an input image size of [batch, 3, 299, 299].

2. Inception models have auxiliary branches which contribute to the loss only during training.  The reported classification loss only considers the main classification loss.

3. Inception_V3 normalizes the input inside the network itself.  More details can be found in @soumendukrg's PR #425 [comments](https://github.com/NervanaSystems/distiller/pull/425#issuecomment-557941736).

NOTE: Training using Inception_V3 is only possible on a single GPU as of now. This issue talks about this problem. I have checked and this problem persists in torch 1.3.0:
[inception_v3 of vision 0.3.0 does not fit in DataParallel of torch 1.1.0 #1048](https://github.com/pytorch/vision/issues/1048

)

Co-authored-by: Neta Zmora <neta.zmora@intel.com>

shorten the TOC

shorten the basic version

Add hsi-toolbox

Added TorchFI citation

This script shows how to specify a compression-schedule directly
using Distiller's API, instead of using a YAML specification

examples/scheduling_api/direct_api_pruning.py

Added masking primitives:
 -mask_tensor
 -create_mask_threshold_criterion
 -create_mask_level_criterion
 -create_mask_sensitivity_criterion

 These APIs have a clearer name and communicate their
 responsibility better: create a tensor mask, based on
 some criterion.  Previously,
 distiller.pruning.create_mask_threshold_criterion was
 named distiller.threshold_mask which did not communicate
 well what this function did.
 Masking functionality is no longer hidden
 inside the Pruner instances, so they can be used directly
 by an application, or to compose new Pruner classes.

Removed file distiller.pruning.pruner:
 -The base-class _ParameterPruner is useless and adds
 needless details to the implementation.

AGP: Separated the pruning-rate schedule from the
 rest of the logic.  This allows us to mix-and-match different
 pruning-rate schedules (just like LR schedulers).

As requested in issue #496

Added some more Distiller citations

Fix ToC
Move "Built With" to the Acknowledgement section

Experiment with layout reformatting: shorten the Community section by making it foldable.

Experiment with layout reformatting: shorten the Community section by making it foldable.

Added citations

change distiller citation as suggested in issue #492

Remove warning regarding Distiller release 0.3 (breaking backward compat)

The gitdb versioning issue is resolved internally in gitpython
3.1.0, so moving to that and removing specific gitb requirements

* BUGFIX: Fixed wrong attribute name for zero-point in conversion
  of eltwise add/mult and concat
* Add PyTorch PTQ convert for embedding (converted to FP32
  embedding + quant op)
* Fix conversion function to work with tuple/list model inputs

* Move image classification specific setup code to separate script at
  examples/classifier_compression/ptq_lapq.py
* Make ptq_coordinate_search function completely independent of
  command line arguments
* Change LAPQ command line args function to update existing
  pre-existing parser (changed CLAs perfix to 'lapq' for more clarity)
* Enable LAPQ from compress_classifier.py (trigger with --qe-lapq)
* Add pointers in documentation

Convert Distiller PTQ models to "native" PyTorch PTQ (#458)

* New API: distiller.quantization.convert_distiller_ptq_model_to_pytorch()
* Can also be called from PostTrainLinearQuantizer instance:
    quantizer.convert_to_pytorch()
* Can also trigger from command line in image classification sample
* Can save/load converted modules via apputils.load/save_checkpoint
* Added Jupyter notebook tutorial

* Converted modules have only the absolutely necessary quant-dequant
  operations. For a fully quantized model, this means just quantization
  of model input and de-quantization of model output. If a user keeps
  specific internal layers in FP32, quant-dequant operations are added
  as needed
* Can configure either 'fbgemm' or 'qnnpack' backend. For 'fbgemm' we
  take care of preventing overflows (aka "reduce_range" in the PyTorch
  API)

"Loss Aware Post-Training Quantization" (Nahshan et al., 2019)

Paper: https://arxiv.org/abs/1911.07190 
Reference implementation:
  https://github.com/ynahshan/nn-quantization-pytorch/tree/master/lapq

Proper documentation is still TODO, for now see the example YAML file
at 'examples/quantization/post_train_quant/resnet18_imagenet_post_train_lapq.yaml'

* Implemented in distiller/quantization/ptq_coordinate_search.py
* At the moment that file both the model-independent algorithm
  implementation and image-classification specific sample script.
  Still TODO: Refactor that

* Post train quantization changes (range_linear):
  * Added getters/setters for quantization parameters (scale/zero_point)
    and clipping values
  * Add option to save backup of FP32 weights to allow re-quantization
    after quantizer was created.
  * Add option to clip weights in addition to activations
  * Fix fusions to not occur only when activations aren't quantized
  * RangeLinearFakeQuantWrapper:
    * Make inputs quantization optional
    * In case of ReLU + ACIQ, clip according to input stats

* Data loaders:
  * Add option to not load train set at all from disk (to speed up
    loading time in post-training runs)
  * Modified "image_classifier.py" accordingly

Temp patch until moving to torchvision 0.5.
See https://github.com/pytorch/vision/issues/1712#issuecomment-575036523

Fix the formatting of a ValueError raised when a module is missing
an attribute, when collecting activation statistics.

(we use 8-bit values below, but this applies to any bit-width)
* We use the notion of "full" and "restricted" quantized range for
  symmetric quantization (see section 2.2 in https://arxiv.org/abs/1806.08342)
* "Full" quantized range ==> [-128, 127], "restircted" ==> [-127, 127]
* Until now, when doing symmetric quantization we assumed a "full"
  range when saturating after quantization, but calculated the scale
  factor as if the range was restricted. This means we weren't making
  full utilization of the quantized range.
* On the other hand, in some other implementations of quantization (e.g.
  TensorFlow), the "restricted" range is used.
* So, we make it an option to use either the proper "full" range
  (q_min = -128) or "restricted" range (q_min = -127).
* LinearQuantMode.SYMMETRIC now means the "full" range is used, and
  added LinearQuantMode.SYMMETRIC_RESTRICTED for using the "restricted"
  range.
* Updated tests and documentation.