../../../distiller/apputils/image_classifier.py – remove unused
`--extras` command-line argument

Odds and ends commit

A bundle of very small, and mostly non-functional, changes to the code.
Mostly they are unrelated to each other
../../../distiller/apputils/checkpoint.py – add info to exception

../../../distiller/apputils/image_classifier.py – remove unused
`--extras` command-line argument

../../../distiller/thinning.py – code refactoring (non-functional)
except for adding a new public API: contract_model()

../../classifier_compression/compress_classifier.py – use
contract_model() when using `--thinnify`

../../lottery_ticket/README.md – remove illegal characters in
the text

ACIQ clipping method, as described in:

Post training 4-bit quantization of convolutional networks for rapid-deployment
(Ron Banner , Yury Nahshan, Daniel Soudry)
(NeurIPS 2019)

https://arxiv.org/abs/1810.05723



Co-authored-by: Yury Nahshan <yury.nahshan@intel.com>
Co-authored-by: Lev Zlotnik <lev.zlotnik@intel.com>

Added explicit thank you mention for Cadene

This patch moves the check on remaining training epochs into the training loop, as it is irrelevant for non-training invocation of ClassifierCompressor.
Thus, allowing resumed checkpoints be evaluated regardless of the epochs limitation.

This patch moves the check on remaining training epochs into the training loop, as it is irrelevant for non-training invocation of ClassifierCompressor.
Thus, allowing resumed checkpoints be evaluated regardless of the epochs limitation.

* The is_simulated_quant_weight_shifted flag was a Python bool,
  modifying of which during forward() isn't supported when 
  DataParallel is used, as per
  https://pytorch.org/docs/stable/nn.html#dataparallel-layers-multi-gpu-distributed
* Instead, registered it as an integer tensor buffer

Integrate the code for the DDPG agent from:
https://github.com/mit-han-lab/amc-release

The instructions for cloning HAN's code and then making changes
to fit Distiller were too complicated, so we added the integrated
files to distiller/examples/auto_compression/amc/rl_lib/hanlab

Mainly: moved NetworkWrapper to a separate file.

FMReconstructionChannelPruner: add support for nn.Linear layers
utils.py: add non_zero_channels()
thinning: support removing channels from FC layers preceding Conv layers
test_pruning.py: add test_row_pruning()
scheduler: init from a dictionary of Maskers
coach_if.py – fix imports of Clipped-PPO and TD3

This command command-line argument allows us to save the
randomly-initialized model before training (useful for lottery-ticket
method).
This commit was accidentally left out of the Lottery-Ticket Hypothesis
commit from Aug 26.

Added the ability to configure the parameter types (weight/bias)
of parameters for which we collect statistics.

For the PythonLogger backend: some logs were sent to
The root-logger instead of the configured module logger.
This made it impossible to use fine-grained control the
output of level of the Python logger.

Added support for saving the randomly initialized network before
starting training; and added an implmentation showing how to extract
a (winning) lottery ticket from the prestine network, and the
pruned network.

Updated documentation per issue wq#359

When adding support for specifying the rounding function for pruning structures, the code of `ActivationRankedFilterPruner` was updated incorrectly (adding an extra argument).
This code is not covered by any of the tests and therefore the bug was not caught in time.  To do: add unit test.

* NCF scripts with Distiller integration

This NCF implementation is based on the implementation found in the MLPerf
Training GitHub repository, specifically on the last revision of the code
before the switch to the extended dataset. See:
https://github.com/mlperf/training/tree/fe17e837ed12974d15c86d5173fe8f2c188434d5/recommendation/pytorch

We've made several modifications to the code:
* Removed all MLPerf specific code including logging
* In ncf.py:
  * Added calls to Distiller compression APIs
  * Added progress indication in training and evaluation flows
* In neumf.py:
  * Added option to split final FC layer
  * Replaced all functional calls with modules so they can be detected
    by Distiller
* In dataset.py:
  * Speed up data loading - On first data will is loaded from CSVs and
    then pickled. On subsequent runs the pickle is loaded. This is much
    faster than the original implementation, but still very slow.
  * Added progress indication during data load process
* Removed some irrelevant content from README.md

* ncf.py - fix wrong nb_users/items

* NCF changes to make it compatible with latest changes in master

* Pass the 'sigmoid' flag in NeuMF.forward as a bool tensor instead of
  a simple boolean. Required to make the model traceable (it?d be better
  to not have it an argument of forward at all, but keeping changes to
  a minimum)
* Call prepare_model with dummy_input

* Cleanup code, update reqs, add stats files, add details to README

* Update NCF README.md

* Remove debug comments

* Remove unnecessary script + NCF QAT scheduler (for now)

This is inspired by @barrh’s PR https://github.com/NervanaSystems/distiller/pull/246
but it at a “slower-integration-pace” and w/o changing APIs.

1. create_model() attaches model attributes (arch, dataset, is_parallel) to created models.
2. save_checkpoint() stores the new model attributes with checkpoint metadata
3. load_checkpoint() can be invoked with model=None, in which case we attempt
to create the model from the stored checkpoint metadata.

Rearranged the code for easier reading and maintenance

When trying to create a model using an unsupported dataset, create_model()
should raise a ValueError, but didn't.
One of the unit-tests (test_create_model_cifar in test_infra.py) was
designed to test this condition (creating a model using an unsupported
dataset - cifar100 in this case), but due to DataParallel implementation
details, the test condition did not function on multi-GPU machines and
did fail the test.
Unit tests should also be executed on single-GPU machines for full-coverage.

This test uses MNIST for faster execution and test various
pruners and their scheduling.

Allow loading a thinned model from a checkpoint file that doesn't
contain a CompressionSchedule.  After thinning, we usually don't have,
or care about, a pruning schedule - so we don't need to force the
user to create one, just for saving in the checkpoint.

init_from_masks_dict() - his is a convenience function to initialize
a CompressionScheduler from a dictionary

we still need/want to refactor further, so that we only
load a dataset (train/valid/test) when we actually need it.
Currently we load all 3 DSs as soon as we need one of them,
as this isn't exactly lazy loading...