"Loss Aware Post-Training Quantization" (Nahshan et al., 2019)

Paper: https://arxiv.org/abs/1911.07190 
Reference implementation:
  https://github.com/ynahshan/nn-quantization-pytorch/tree/master/lapq

Proper documentation is still TODO, for now see the example YAML file
at 'examples/quantization/post_train_quant/resnet18_imagenet_post_train_lapq.yaml'

* Implemented in distiller/quantization/ptq_coordinate_search.py
* At the moment that file both the model-independent algorithm
  implementation and image-classification specific sample script.
  Still TODO: Refactor that

* Post train quantization changes (range_linear):
  * Added getters/setters for quantization parameters (scale/zero_point)
    and clipping values
  * Add option to save backup of FP32 weights to allow re-quantization
    after quantizer was created.
  * Add option to clip weights in addition to activations
  * Fix fusions to not occur only when activations aren't quantized
  * RangeLinearFakeQuantWrapper:
    * Make inputs quantization optional
    * In case of ReLU + ACIQ, clip according to input stats

* Data loaders:
  * Add option to not load train set at all from disk (to speed up
    loading time in post-training runs)
  * Modified "image_classifier.py" accordingly

Temp patch until moving to torchvision 0.5.
See https://github.com/pytorch/vision/issues/1712#issuecomment-575036523

Fix the formatting of a ValueError raised when a module is missing
an attribute, when collecting activation statistics.

(we use 8-bit values below, but this applies to any bit-width)
* We use the notion of "full" and "restricted" quantized range for
  symmetric quantization (see section 2.2 in https://arxiv.org/abs/1806.08342)
* "Full" quantized range ==> [-128, 127], "restircted" ==> [-127, 127]
* Until now, when doing symmetric quantization we assumed a "full"
  range when saturating after quantization, but calculated the scale
  factor as if the range was restricted. This means we weren't making
  full utilization of the quantized range.
* On the other hand, in some other implementations of quantization (e.g.
  TensorFlow), the "restricted" range is used.
* So, we make it an option to use either the proper "full" range
  (q_min = -128) or "restricted" range (q_min = -127).
* LinearQuantMode.SYMMETRIC now means the "full" range is used, and
  added LinearQuantMode.SYMMETRIC_RESTRICTED for using the "restricted"
  range.
* Updated tests and documentation.

* Fake quant wrapper now also works on (fake) quantized inputs
* Remove 'requires_quantized_inputs' flag
* Unrelated: Moved LinearQuantMode enum to q_utils

In PostTrainLinearQuantizer and QuantAwareTrainRangeLinearQuantizer

Add directionality to SummaryActivationStatsCollector to allow collection of statistics on incoming and outgoing activations/feature-maps; instead of just outgoing activations.

Also includes some code refactoring.

* Limit batch size to 128 when initiating from image classification app
* Don't raise inplace error in case of Dropout module

* Make it easier to find sample apps for different workload types
* Add READMEs for sample apps the didn't have any
* Update readmes with experiment results where applicable

Added tables of results for 85% sparsity

* Weights-only PTQ:
  * Allow RangeLinearQuantWrapper to accept num_bits_acts = None, in
    which case it'll act as a simple pass-through during forward
  * In RangeLinearQuantParamLayerWrapper, if bits_activations is None
    and num_bits_params > 0, Perform quant and de-quant of the
    parameters instead of just quant.
* Activations-only PTQ:
  * Enable activations only quantization for conv/linear modules. When
    PostTrainLinearQuantizer detects # bits != None for activations 
    and # bits == None for weights, a fake-quantization wrapper will
    be used.
* Allow passing 0 in the `--qe-bits-acts` and `--qe-bits-wts` command
  line arguments to invoke weights/activations-only quantization,
  respectively.
* Minor refactoring for clarity in PostTrainLinearQuantizer's replace_*
  functions

Results changed following commit 9e7ef987 (#402)

compute-summary and png-summary currently work with image classifiers
only.

When multi-processing, we want only one process to generate the
summary, while the other processes do nothing (lazy bums!)

Add an example of compressing OD pytorch models.

In this example we compress torchvision's object detection models - FasterRCNN / MaskRCNN / KeypointRCNN.
We've modified the reference code for object detection to allow easy compression scheduling with YAML configuration.

of a model by name relative to the root of the model.

- define ALMOST_ONE
- define op_type
- remove sanity assert (need to understand what tolerance value to use
in the assert)

Co-authored-by: csc12138
Co-authored-by: wangyidong3

- define ALMOST_ONE
- define op_type
- remove sanity assert (need to understand what tolerance value to use
in the assert)

This will help define and use different performance sorting schemes.
E.g. this will address the issue raised in issue #411

Small variances can occur when using different cudnn versions,
even when the environment and distiller version is the same.

Said commit was wrong: the default inializations in pytorch are not
the same as in our code.  For example, the default convolution
weight initialization uses Kaiming-uniform, while we used
Kaiming-normal.
For backward comparability of the model behavior, we need to
revert to the old behavior.
This reverts commit 6913687f.

Update the definition of the exits using info from Haim.

This is still very unsatsifactory because we don't have working
examples to show users :-(

except for the case of VGG, our parameter initialization code was
matched the default pytorch initialization (per torch.nn operation),
so writing the initialization code ourselves can only lead to
more code and maintenance; and also we would not benefit from
improvements that occur at the pytorch level (e.g. if FB finds a
better initialization for nn.conv2d than today's kaiming init, we
would not benefit).
The VGG initialization we had was "suspicious" and so reverting
to the default seems reasonable.

* summary_graph.py:
  * Change ONNX op.uniqueName() to op.debugName()
  * Removed scope-naming workaround which isn't needed in PyTorch 1.3
* Tests:
  * Naming of trace entries changed in 1.3. Fixed SummaryGraph unit
    test that checked that
  * Adjusted expected values in full_flow_tests
  * Adjusted tolerance in test_sim_bn_fold
  * Filter some new warnings

Prevent exception when loading checkpoints from a home directory

Two strings represented the library version: one in distiller.__init__.py
and one in setup.py.
This can lead to two difference version string values.
The fix: have distiller.__init__.py read the version string from the
package installation.
This assumes that we've installed distiller properly, but we've been
making this assumption for a long time in our code (e.g. how we do
imports of distiller from the `tests` directory).