This notebook shows an example of how to generate activation histograms for a specific model and dataset.
## But I Already Know How To Generate Histograms...
If you already generated histograms using Distiller outside this notebook, you can still use it to visualize the data:
* To load the raw data saved by Distiller and visualize it, go to [this section](#Plot-Histograms)
* If enabled saving histogram images and want to view them, go to [this section](#Load-Histogram-Images-from-Disk)
%% Cell type:code id: tags:
``` python
%matplotlibinline
importtorch
importmatplotlib.pyplotasplt
importos
importmath
importtorchnetastnt
fromipywidgetsimportwidgets,interact
importdistiller
fromdistiller.modelsimportcreate_model
device=torch.device('cuda')
# device = torch.device('cpu')
# Load some common code and configure logging
# We do this so we can see the logging output coming from
# Distiller function calls
%run'./distiller_jupyter_helpers.ipynb'
msglogger=config_notebooks_logger()
```
%% Cell type:markdown id: tags:
## Load Your Model
For this example we'll use a pre-trained image classification model.
### Note on Parallelism
Currently, Distiller's implementation of activations histograms collection does not accept models which contain [`DataParallel`](https://pytorch.org/docs/stable/nn.html?highlight=dataparallel#torch.nn.DataParallel) modules. So here we create the model without parallelism to begin with. If you have a model which includes `DataParallel` modules (for example, if loaded from a checkpoint), use the following utlity function to convert the model to serialized execution:
model=model.to(device)# Comment out if not applicable
```
%% Cell type:markdown id: tags:
## Prepare Data
Usually it is not required to collect histograms based on the entire dataset, and only a representative subset is used (that also helps reduce the runtime).
***Subset size:** There is no golden rule for selecting the size of the subset. Anywhere between 1-10% of the validation/test set should work.
***Representative data:** Whatever size is chosen, it is important to make sure that the subset is selected in a way that covers as much of the distribution of the data as possible. So, for example, if the dataset is organized by classes by default, we should make sure to select items randomly and not in order.
**Note:** Working on only a subset of the data can be taken care of at data preparation time, or it can be delayed to the actual model evaluation function (for example, executing only a specific number of mini-batches). In this example we take care of it during data preparation.
%% Cell type:code id: tags:
``` python
# We use Distiller's built-in data loading functionality for ImageNet,
# which takes care of randomizing the data before selecting the subset.
# While it creates train, validation and test data loaders, we're only
# interested in the test dataset in this example.
#
# Subset size: Here we'll go with 1% of the test set, mostly for the
# sake of speed. We control this with the 'effective_test_size' argument.
#
# We set the 'fixed_subset' argument to make sure we're using the
# same subset for both phases of histogram collection - more on that below
We define a fairly bare-bones evaluation function. Recording the loss and accuracy isn't strictly necessary for histogram collection. We record them nonetheless, so we can verify the data subset being used achieves results that are on par from what we'd expect from a representative subset.
Histogram collection is implemented using Distiller's "Collector" mechanism, specifically in the `ActivationHistogramsCollector` class. It is stats-based, meaning it requires pre-computed min/max values per-tensor to be provided.
The min/max stats are expected as a dictionary with the following structure:
```YAML
'layer_name':
'inputs':
0:
'min': value
'max': value
...
n:
'min': value
'max': value
'output':
'min': value
'max': value
```
Where n is the number of inputs the layer has. The `QuantCalibrationStatsCollector` collector class generates stats in the required format.
To streamline this process, a utility function is provided: `distiller.data_loggers.collect_histograms`. Given a model and a test function, it will perform the required stats collection followed by histograms collection. If the user has already computed min/max stats beforehand, those can provided as a dict or as a path to a YAML file (as saved by `QuantCalibrationStatsCollector`). In that case, the stats collection pass will be skipped.
### Dataset Perparation in Context of Stats-Based Histograms
If the data used for min/max stats collection is not the same as the data used for histogram collection, it is highly likely that when collecting histograms some values will fall outside the pre-calculated min/max range. When that happens, the value is **clamped**. Assuming the subsets of data used in both cases are representative enough, this shouldn't have a major effect on the results.
One can choose to avoid this issue by making sure we use the same subset of data in both passes. How to make sure of that will, of course, differ from one use case to another. In this example we do this by using the enabling `fixed_subset` flag when calling `load_data` above.
%% Cell type:code id: tags:
``` python
# The test function passed to 'collect_histograms' must have an
# argument named 'model' which accepts the model for which histograms
# are to be collected. 'collect_histograms' will not set any other
# arguments.
# We'll use Python's 'partial' to handle the set the rest of the
# arguments for the test function before calling 'collect_histograms'
If you enabled saving of histogram images above, or have images from a collection executed externally, you can use the code below to display the images.
# Convert Distiller Post-Train Quantization Models to "Native" PyTorch
## Background
As of version 1.3 PyTorch comes with built-in quantization functionality. Details are available [here](https://pytorch.org/docs/stable/quantization.html). Distiller's and PyTorch's implementations are completely unrelated. An advantage of PyTorch built-in quantization is that it offers optimized 8-bit execution on CPU and export to GLOW. PyTorch doesn't offer optimized 8-bit execution on GPU (as of version 1.4).
At the moment we are still keeping Distiller's separate API and implementation, but we've added the capability to convert a **post-training quantization** model created in Distiller to a "Distiller-free" model, comprised entirely of PyTorch built-in quantized modules.
Distiller's quantized layers are actually simulated in FP32. Hence, comparing a Distiller model running on CPU to a PyTorch built-in model, the latter will be significantly faster on CPU. However, a Distiller model on a GPU is still likely to be faster compared to a PyTorch model on CPU. So experimenting with Distiller and converting to PyTorch in the end could be useful. Milage may vary of course, depending on the actual HW setup.
Let's see how the conversion works.
%% Cell type:code id: tags:
``` python
importtorch
importmatplotlib.pyplotasplt
importos
importmath
importtorchnetastnt
fromipywidgetsimportwidgets,interact
fromcopyimportdeepcopy
fromcollectionsimportOrderedDict
importdistiller
fromdistiller.modelsimportcreate_model
importdistiller.quantizationasquant
# Load some common code and configure logging
# We do this so we can see the logging output coming from
# Distiller function calls
%run'./distiller_jupyter_helpers.ipynb'
msglogger=config_notebooks_logger()
```
%% Cell type:markdown id: tags:
## Create Model
%% Cell type:code id: tags:
``` python
# By default, the model is moved to the GPU and parallelized (wrapped with torch.nn.DataParallel)
# If no GPU is available, a non-parallel model is created on the CPU
1. Distiller takes care of quantizing the inputs within the quantized modules PyTorch quantized modules assume the input is already quantized. Hence, for cases where a module's input is not quantized, we explicitly add a quantization operation for the input. The first layer in the model, `conv1` in ResNet18, is such a case
2. Both Distiller and native PyTorch support fused ReLU. In Distiller, this is somewhat obscurely indicated by the `clip_half_range` attribute inside `output_quant_settings`. In PyTorch, the module type is explicitly `QuantizedConvReLU2d`.
Example of internal layers which don't require explicit input quantization:
%% Cell type:code id: tags:
``` python
print('layer1.0.conv1')
print(pyt_model.layer1[0].conv1)
print('\nlayer1.0.add')
print(pyt_model.layer1[0].add)
```
%% Cell type:markdown id: tags:
### Automatic de-quantization <--> quantization in the model
For each quantized module in the Distiller implementation, we quantize the input and de-quantize the output.
So, if the user explicitly sets "internal" modules to run in FP32, this is transparent to the other quantized modules (at the cost of redundant quant-dequant operations).
When converting to PyTorch we remove these redundant operations, and keep just the required ones in case the user explicitly decided to run some modules in FP32.
For an example, consider a ResNet "basic block" with a residual connection that contains a downsampling convolution. Let's see how such a block looks in our fully-quantized, converted model:
%% Cell type:code id: tags:
``` python
print(pyt_model.layer2[0])
```
%% Cell type:markdown id: tags:
We can see all layers are either built-in quantized PyTorch modules, or identity operations representing fused operations. The entire block is quantized, so we don't see any quant-dequnt operations in the middle.
Now let's create a new quantized model, and this time leave the 'downsample' module in FP32:
1. The `downsample` module now contains a de-quantize op before the actual convolution
2. The `add` module now contains a quantize op before the actual add. Note that the add operation accepts 2 inputs. In this case the first input (index 0) comes from the `conv2` module, which is quantized. The second input (index 1) comes from the `downsample` module, which we kept in FP32. So, we only need to quantized the input at index 1. We can see this is indeed what is happening, by looking at the `ModuleDict` inside the `quant` module, and noticing it has only a single key for index "1".
Let's see how the `add` module would look if we also kept the `conv2` module in FP32:
We can see that now both inputs to the add module are being quantized.
%% Cell type:markdown id: tags:
## Another API for Conversion
In some cases we don't have the actual quantizer. For example - if the Distiller quantized module was loaded from a checkpoint. In those cases we can call a `distiller.quantization` module-level function (In fact, the Quantizer method we used earlier is a wrapper around this function).
