Test and Benchmarks

Directory Organization

The hpvm/test directory holds all tests and benchmarks in HPVM and is organized as follows:

• hpvm_pass/: unit and regression tests for HPVM Passes, written in LLVM-bitcode.

• hpvm-tensor-rt/: unit tests for the HPVM tensor_runtime. This folder just contains the test fixtures and the test files to run. The actual test cases live under ${project_root}/hpvm/projects/hpvm-tensor-rt/tests/.

• benchmarks/: includes a few applications written in HPVM-C, a template, and directions for compiling and running these benchmarks.

  • benchmarks/parboil: Selected benchmarks from the Parboil benchmark suite.
  • benchmarks/pipeline: An edge detection pipeline benchmark.
  • benchmarks/hpvm-cava: A Camera ISP pipeline, adapted from C code provided from our collaborators at Harvard.

• dnn_benchmarks/: ten (10) DNN benchmarks in HPVM-C, Keras and PyTorch, supported by ApproxHPVM. This tests HPVM as well as the Keras and PyTorch frontends.

  • dnn_benchmarks/hpvm-c contains the HPVM-C version of these DNNs. Their organization and usage are similar to the benchmarks under benchmarks/.

    Each subfolder contains a DNN with 2 versions (2 .cpp files): the tensor-targeted version which compiles to tensor_runtime, and the cudnn-targeted version which compiles to operators in cuDNN (has _cudnn in name).

  • dnn_benchmarks/keras contains these DNNs implemented in Keras, and code for generating them down to HPVM-C (testing Keras frontend).

  • dnn_benchmarks/pytorch contains these DNNs in PyTorch and code for generating them down to HPVM-C (testing PyTorch/ONNX frontend).

    • ./dnn is a local package with these 10 DNNs implemented in PyTorch as examples. This package is not installed with HPVM.
    • ./test_frontend contains tests on inference accuracy of code generated by the PyTorch frontend.
    • ./test_{profiling|tuning} contains tests on performing profiling/tuning on frontend-generated binary.

  • dnn_benchmarks/tensor-rt-src contains these DNNs directly implemented in tensor_runtime functions. These are for reference purpose only and not actively used in the HPVM system or testing.

Running Test Cases and Benchmarks

The easiest way to run tests is to use make targets, which will also take care of all compilation of test cases and test fixtures. The following targets runs these tests respectively:

• make -j check-hpvm-pass runs tests in hpvm_pass: hpvm_pass/**/*.ll. These are regression and unit tests for HPVM passes.

• make -j check-hpvm-tensor-rt checks the approximation implementations of tensor_runtime.

• make -j check-hpvm-dnn runs all 20 DNN benchmarks under dnn_benchmarks/hpvm-c (10 DNNs x 2 versions) and validates their accuracy.

  Note that this can take quite long due to the size of DNNs and datasets. Depending on your hardware capability, this test can take 5-30 minutes. Also, this is set to run sequentially out of GPU memory concerns.

• make -j check-hpvm-keras-acc generates all 10 DNNs with Keras frontend, runs them and checks their accuracy. This tests the Keras frontend in isolation.

• Similarly, make -j check-hpvm-torch-acc generates all 10 DNNs with PyTorch frontend, runs them and checks their accuracy, to test the PyTorch frontend in isolation.

• make -j check-hpvm-torch-tuning runs predtuner with binaries from PyTorch frontend to exercise both empirical and predictive autotuning. This is only done for a few smaller networks for 5 iterations, as it is extremely time-consuming.

• make -j check-hpvm-torch-profiling runs hpvm-profiler with binaries from PyTorch frontend, and presents the tradeoff curve with profiled speedup. This is only done for a few smaller networks.

Underneath, llvm-lit is used to discover and run the tests.

Compiling Benchmarks

This section explains how to compile the benchmarks without running them as tests.

HPVM-C DNN Benchmarks

To build (not run) all dnn_benchmarks/hpvm-c, use make -j dnn_benchmarks. For each benchmark ${bench_name}, the binary is generated at ${build_dir}/tools/hpvm/test/dnn_benchmarks/hpvm-c/${bench_name}.

Alternatively, it's possible to build just 1 DNN benchmark. The output of CMake shows a list of these benchmarks as target names, starting with

List of test dnn benchmarks: alexnet2_cifar10;alexnet2_cifar10...

Currently, there are 20 of them. These are:

_cudnn suffix indicates the code is generated onto cuDNN functions. Otherwise they are generated to tensor_runtime DNN functions which are hand-written in CUDA.

Other HPVM-C Benchmarks

There are 6 benchmarks under benchmarks/: hpvm-cava and pipeline are single benchmarks, while parboil/ is a collection of 4 benchmarks.

To build hpvm-cava or pipeline, use make -j hpvm_cava_{cpu|gpu} or make -j pipeline_{cpu|gpu}. The cpu or gpu suffix indicates the device the kernels in the benchmark run on. For hpvm-cava, the binary is generated under ${build_dir}/tools/hpvm/test/benchmarks/hpvm-cava, while pipeline binaries are under ${build_dir}/tools/hpvm/test/benchmarks/pipeline.

The parboil benchmarks are only available through Makefile. We will move them into CMake in the next release.