Newer
Older
Test and Benchmarks
========================
Directory Organization
----------------------
The `hpvm/test` directory holds all tests and benchmarks in HPVM and is organized as follows:
*
``hpvm_pass/``: unit and regression tests for HPVM Passes, written in LLVM-bitcode.
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
*
``benchmarks/``: includes a few applications written in HPVM-C, a template, and directions for compiling and running these benchmarks.
* ``benchmarks/parboil``: Selected benchmarks from the `Parboil <http://impact.crhc.illinois.edu/parboil/parboil.aspx>`_ benchmark suite.
* ``benchmarks/pipeline``: An edge detection pipeline benchmark.
* ``benchmarks/hpvm-cava``: A Camera ISP pipeline, adapted from C code provided from our collaborators at `Harvard <http://vlsiarch.eecs.harvard.edu>`_.
*
``dnn_benchmarks/``: ten (10) DNN benchmarks in HPVM-C, Keras and PyTorch, supported by ApproxHPVM.
This tests HPVM as well as the Keras and PyTorch frontends.
*
``dnn_benchmarks/hpvm-c`` contains the HPVM-C version of these DNNs.
Their organization and usage are similar to the benchmarks under ``benchmarks/``.
Each subfolder contains a DNN with 2 versions (2 ``.cpp`` files):
the ``tensor``-targeted version which compiles to ``tensor_runtime``,
and the ``cudnn``-targeted version which compiles to operators in ``cuDNN``
(has ``_cudnn`` in name).
*
``dnn_benchmarks/keras`` contains these DNNs implemented in Keras,
and code for generating them down to HPVM-C (testing Keras frontend).
* ``dnn_benchmarks/pytorch`` contains these DNNs in PyTorch
and code for generating them down to HPVM-C (testing PyTorch/ONNX frontend).
The code generated from Keras and PyTorch frontend should be largely similar and functionally equivalent.
Running Test Cases and Benchmarks
---------------------------------
The easiest way to run tests is to use ``make`` targets,
which will also take care of all compilation of test cases and test fixtures.
The following targets runs these tests respectively:
* ``make -j check-hpvm-pass`` runs tests in ``hpvm_pass``: ``hpvm_pass/**/*.ll``.
These are regression and unit tests for HPVM passes.
*
``make -j check-hpvm-dnn`` runs all 20 DNN benchmarks under ``dnn_benchmarks/hpvm-c``
(10 DNNs x 2 versions) and validates their accuracy.
*Note* that this can take quite long due to the size of DNNs and datasets.
Depending on your hardware capability, this test can take 5-30 minutes.
Also, this is set to run sequentially out of GPU memory concerns.
*
``make -j check-hpvm-profiler`` runs ``hpvm-profiler`` on some smaller networks
(as it is extremely time-consuming) and presents the tradeoff curve with profiled speedup.
*Note* that if you're on an NVIDIA Jetson TX2, you may want to run
``bash dnn_benchmarks/profiling/jetson_clocks.sh``
to ensure that the clocks are running at the maximum frequency
Underneath, ``llvm-lit`` is used to discover and run the tests.
``benchmarks/`` can only be compiled in-source with ``make``.
We are working to migrate it into the ``cmake`` system.
Compiling Benchmarks
--------------------
This section explains how to compile the benchmarks without running them as tests.
HPVM-C DNN Benchmarks
^^^^^^^^^^^^^^^^^^^^^
To build (not run) all ``dnn_benchmarks/hpvm-c``, use ``make -j dnn_benchmarks``.
For each benchmark ``${bench_name}``, the binary is generated at
``${build_dir}/tools/hpvm/test/dnn_benchmarks/hpvm-c/${bench_name}``.
Alternatively, it's possible to build just 1 DNN benchmark.
The output of CMake shows a list of these benchmarks as target names, starting with
..
List of test dnn benchmarks: alexnet2_cifar10;alexnet2_cifar10...
Currently, there are 20 of them. These are:
.. list-table::
:header-rows: 1
* -
-
* - lenet_mnist
- lenet_mnist_cudnn
* - alexnet_cifar10
- alexnet_cifar10_cudnn
* - alexnet2_cifar10
- alexnet2_cifar10_cudnn
* - vgg16_cifar10
- vgg16_cifar10_cudnn
* - vgg16_cifar100
- vgg16_cifar100_cudnn
* - mobilenet_cifar10
- mobilenet_cifar10_cudnn
* - resnet18_cifar10
- resnet18_cifar10_cudnn
* - alexnet_imagenet
- alexnet_imagenet_cudnn
* - vgg16_imagenet
- vgg16_imagenet_cudnn
* - resnet50_imagenet
- resnet50_imagenet_cudnn
``_cudnn`` suffix indicates the code is generated onto cuDNN functions.
Otherwise they are generated to ``tensor_runtime`` DNN functions which are hand-written in CUDA.