Test and Benchmarks
========================
Directory Organization
----------------------
The ``hpvm/test`` directory holds all tests and benchmarks in HPVM and is organized as follows:
* ``hpvm_pass/``: unit and regression tests for HPVM Passes, written in LLVM-bitcode.
* ``hpvm-tensor-rt/``: unit tests for the HPVM `tensor_runtime`.
This folder just contains the test fixtures and the test files to run.
The actual test cases live under ``${project_root}/hpvm/projects/hpvm-tensor-rt/tests/``.
* ``benchmarks/``: includes a few applications written in HPVM-C, a template, and directions for compiling and running these benchmarks.
* ``benchmarks/parboil``: Selected benchmarks from the `Parboil <http://impact.crhc.illinois.edu/parboil/parboil.aspx>`_ benchmark suite.
* ``benchmarks/pipeline``: An edge detection pipeline benchmark.
* ``benchmarks/hpvm-cava``: A Camera ISP pipeline, adapted from C code provided from our collaborators at `Harvard <http://vlsiarch.eecs.harvard.edu>`_.
* ``dnn_benchmarks/``: ten (10) DNN benchmarks in HPVM-C, Keras and PyTorch, supported by ApproxHPVM.
This tests HPVM as well as the Keras and PyTorch frontends.
*
``dnn_benchmarks/hpvm-c`` contains the HPVM-C version of these DNNs.
Their organization and usage are similar to the benchmarks under ``benchmarks/``.
Each subfolder contains a DNN with 2 versions (2 ``.cpp`` files):
the ``tensor``-targeted version which compiles to `tensor_runtime`,
and the ``cudnn``-targeted version which compiles to operators in ``cuDNN``
(has ``_cudnn`` in name).
* ``dnn_benchmarks/keras`` contains these DNNs implemented in Keras,
and code for generating them down to HPVM-C (testing Keras frontend).
* ``dnn_benchmarks/pytorch`` contains these DNNs in PyTorch
and code for generating them down to HPVM-C (testing PyTorch/ONNX frontend).
* ``./dnn`` is a local package with these 10 DNNs implemented in PyTorch as examples.
This package is not installed with HPVM.
* ``./test_frontend`` contains tests on inference accuracy of code generated by the PyTorch frontend.
* ``./test_{profiling|tuning}`` contains tests on performing profiling/tuning
on frontend-generated binary.
* ``dnn_benchmarks/tensor-rt-src`` contains these DNNs directly implemented in `tensor_runtime`
functions. These are for reference purpose only and not actively used in the HPVM system or testing.
Running Test Cases and Benchmarks
---------------------------------
The easiest way to run tests is to use ``make`` targets,
which will also take care of all compilation of test cases and test fixtures.
The following targets runs these tests respectively:
* ``make -j check-hpvm-pass`` runs tests in ``hpvm_pass``: ``hpvm_pass/**/*.ll``.
These are regression and unit tests for HPVM passes.
* ``make -j check-hpvm-tensor-rt`` checks the approximation implementations of `tensor_runtime`.
* ``make -j check-hpvm-dnn`` runs all 20 DNN benchmarks under ``dnn_benchmarks/hpvm-c``
(10 DNNs x 2 versions) and validates their accuracy.
**Note** that this can take quite long due to the size of DNNs and datasets.
Depending on your hardware capability, this test can take 5-30 minutes.
Also, this is set to run sequentially out of GPU memory concerns.
* ``make -j check-hpvm-keras-acc`` generates all 10 DNNs with Keras frontend,
runs them and checks their accuracy. This tests the Keras frontend in isolation.
* Similarly, ``make -j check-hpvm-torch-acc`` generates all 10 DNNs with PyTorch frontend,
runs them and checks their accuracy, to test the PyTorch frontend in isolation.
* ``make -j check-hpvm-torch-tuning`` runs `predtuner` with binaries from PyTorch frontend
to exercise both empirical and predictive autotuning.
This is only done for a few smaller networks for 5 iterations,
as it is extremely time-consuming.
* ``make -j check-hpvm-torch-profiling`` runs `hpvm-profiler` with binaries from PyTorch frontend,
and presents the tradeoff curve with profiled speedup.
This is only done for a few smaller networks.
Underneath, ``llvm-lit`` is used to discover and run the tests.
Compiling Benchmarks
--------------------
This section explains how to compile the benchmarks without running them as tests.
HPVM-C DNN Benchmarks
^^^^^^^^^^^^^^^^^^^^^
To build (not run) all ``dnn_benchmarks/hpvm-c``, use ``make -j dnn_benchmarks``.
For each benchmark ``${bench_name}``, the binary is generated at
``${build_dir}/tools/hpvm/test/dnn_benchmarks/hpvm-c/${bench_name}``.
Alternatively, it's possible to build just 1 DNN benchmark.
The output of CMake shows a list of these benchmarks as target names, starting with
.. code-block:: text
List of test dnn benchmarks: alexnet2_cifar10;alexnet2_cifar10...
Currently, there are 20 of them. These are:
.. list-table::
:header-rows: 1
* -
-
* - lenet_mnist
- lenet_mnist_cudnn
* - alexnet_cifar10
- alexnet_cifar10_cudnn
* - alexnet2_cifar10
- alexnet2_cifar10_cudnn
* - vgg16_cifar10
- vgg16_cifar10_cudnn
* - vgg16_cifar100
- vgg16_cifar100_cudnn
* - mobilenet_cifar10
- mobilenet_cifar10_cudnn
* - resnet18_cifar10
- resnet18_cifar10_cudnn
* - alexnet_imagenet
- alexnet_imagenet_cudnn
* - vgg16_imagenet
- vgg16_imagenet_cudnn
* - resnet50_imagenet
- resnet50_imagenet_cudnn
``_cudnn`` suffix indicates the code is generated onto cuDNN functions.
Otherwise they are generated to `tensor_runtime` DNN functions which are hand-written in CUDA.
Other HPVM-C Benchmarks
^^^^^^^^^^^^^^^^^^^^^^^
There are 6 benchmarks under ``benchmarks/``:
``hpvm-cava`` and ``pipeline`` are single benchmarks, while ``parboil/`` is a collection of 4 benchmarks.
To build ``hpvm-cava`` or ``pipeline``,
use ``make -j hpvm_cava_{cpu|gpu}`` or ``make -j pipeline_{cpu|gpu}``.
The cpu or gpu suffix indicates the device the kernels in the benchmark run on.
For ``hpvm-cava``, the binary is generated under
``${build_dir}/tools/hpvm/test/benchmarks/hpvm-cava``,
while pipeline binaries are under ``${build_dir}/tools/hpvm/test/benchmarks/pipeline``.
The parboil benchmarks are only available through Makefile.
We will move them into CMake in the next release.