Newer
Older
Test and Benchmarks
========================
Directory Organization
----------------------
The ``hpvm/test`` directory holds all tests and benchmarks in HPVM and is organized as follows:
* ``hpvm_pass/``: unit and regression tests for HPVM Passes, written in LLVM-bitcode.
* ``hpvm-tensor-rt/``: unit tests for the HPVM `tensor_runtime`.
This folder just contains the test fixtures and the test files to run.
The actual test cases live under ``${project_root}/hpvm/projects/hpvm-tensor-rt/tests/``.
* ``benchmarks/``: includes a few applications written in HPVM-C, a template, and directions for compiling and running these benchmarks.
* ``benchmarks/parboil``: Selected benchmarks from the `Parboil <http://impact.crhc.illinois.edu/parboil/parboil.aspx>`_ benchmark suite.
* ``benchmarks/pipeline``: An edge detection pipeline benchmark.
* ``benchmarks/hpvm-cava``: A Camera ISP pipeline, adapted from C code provided from our collaborators at `Harvard <http://vlsiarch.eecs.harvard.edu>`_.
* ``dnn_benchmarks/``: ten (10) DNN benchmarks in HPVM-C, Keras and PyTorch, supported by ApproxHPVM.
This tests HPVM as well as the Keras and PyTorch frontends.
*
``dnn_benchmarks/hpvm-c`` contains the HPVM-C version of these DNNs.
Their organization and usage are similar to the benchmarks under ``benchmarks/``.
Each subfolder contains a DNN with 2 versions (2 ``.cpp`` files):
the ``tensor``-targeted version which compiles to `tensor_runtime`,
and the ``cudnn``-targeted version which compiles to operators in ``cuDNN``
(has ``_cudnn`` in name).
* ``dnn_benchmarks/keras`` contains these DNNs implemented in Keras,
and code for generating them down to HPVM-C (testing Keras frontend).
* ``dnn_benchmarks/pytorch`` contains these DNNs in PyTorch
and code for generating them down to HPVM-C (testing PyTorch/ONNX frontend).
* ``./dnn`` is a local package with these 10 DNNs implemented in PyTorch as examples.
This package is not installed with HPVM.
* ``./test_frontend`` contains tests on inference accuracy of code generated by the PyTorch frontend.
* ``./test_{profiling|tuning}`` contains tests on performing profiling/tuning
on frontend-generated binary.
* ``dnn_benchmarks/tensor-rt-src`` contains these DNNs directly implemented in `tensor_runtime`
functions. These are for reference purpose only and not actively used in the HPVM system or testing.
Running Test Cases and Benchmarks
---------------------------------
The easiest way to run tests is to use ``make`` targets,
which will also take care of all compilation of test cases and test fixtures.
The following targets runs these tests respectively:
* ``make -j check-hpvm-pass`` runs tests in ``hpvm_pass``: ``hpvm_pass/**/*.ll``.
These are regression and unit tests for HPVM passes.
* ``make -j check-hpvm-tensor-rt`` checks the approximation implementations of `tensor_runtime`.
* ``make -j check-hpvm-dnn`` runs all 20 DNN benchmarks under ``dnn_benchmarks/hpvm-c``
(10 DNNs x 2 versions) and validates their accuracy.
**Note** that this can take quite long due to the size of DNNs and datasets.
Depending on your hardware capability, this test can take 5-30 minutes.
Also, this is set to run sequentially out of GPU memory concerns.
* ``make -j check-hpvm-keras-acc`` generates all 10 DNNs with Keras frontend,
runs them and checks their accuracy. This tests the Keras frontend in isolation.
* Similarly, ``make -j check-hpvm-torch-acc`` generates all 10 DNNs with PyTorch frontend,
runs them and checks their accuracy, to test the PyTorch frontend in isolation.
* ``make -j check-hpvm-torch-tuning`` runs `predtuner` with binaries from PyTorch frontend
to exercise both empirical and predictive autotuning.
This is only done for a few smaller networks for 5 iterations,
as it is extremely time-consuming.
* ``make -j check-hpvm-torch-profiling`` runs `hpvm-profiler` with binaries from PyTorch frontend,
and presents the tradeoff curve with profiled speedup.
This is only done for a few smaller networks.
Underneath, ``llvm-lit`` is used to discover and run the tests.
Compiling Benchmarks
--------------------
This section explains how to compile the benchmarks without running them as tests.
HPVM-C DNN Benchmarks
^^^^^^^^^^^^^^^^^^^^^
To build (not run) all ``dnn_benchmarks/hpvm-c``, use ``make -j dnn_benchmarks``.
For each benchmark ``${bench_name}``, the binary is generated at
``${build_dir}/tools/hpvm/test/dnn_benchmarks/hpvm-c/${bench_name}``.
Alternatively, it's possible to build just 1 DNN benchmark.
The output of CMake shows a list of these benchmarks as target names, starting with
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
List of test dnn benchmarks: alexnet2_cifar10;alexnet2_cifar10...
Currently, there are 20 of them. These are:
.. list-table::
:header-rows: 1
* -
-
* - lenet_mnist
- lenet_mnist_cudnn
* - alexnet_cifar10
- alexnet_cifar10_cudnn
* - alexnet2_cifar10
- alexnet2_cifar10_cudnn
* - vgg16_cifar10
- vgg16_cifar10_cudnn
* - vgg16_cifar100
- vgg16_cifar100_cudnn
* - mobilenet_cifar10
- mobilenet_cifar10_cudnn
* - resnet18_cifar10
- resnet18_cifar10_cudnn
* - alexnet_imagenet
- alexnet_imagenet_cudnn
* - vgg16_imagenet
- vgg16_imagenet_cudnn
* - resnet50_imagenet
- resnet50_imagenet_cudnn
``_cudnn`` suffix indicates the code is generated onto cuDNN functions.
Otherwise they are generated to `tensor_runtime` DNN functions which are hand-written in CUDA.
Other HPVM-C Benchmarks
^^^^^^^^^^^^^^^^^^^^^^^
There are 6 benchmarks under ``benchmarks/``:
``hpvm-cava`` and ``pipeline`` are single benchmarks, while ``parboil/`` is a collection of 4 benchmarks.
To build ``hpvm-cava`` or ``pipeline``,
use ``make -j hpvm_cava_{cpu|gpu}`` or ``make -j pipeline_{cpu|gpu}``.
The cpu or gpu suffix indicates the device the kernels in the benchmark run on.
For ``hpvm-cava``, the binary is generated under
``${build_dir}/tools/hpvm/test/benchmarks/hpvm-cava``,
while pipeline binaries are under ``${build_dir}/tools/hpvm/test/benchmarks/pipeline``.
The parboil benchmarks are only available through Makefile.
We will move them into CMake in the next release.