Skip to content
Snippets Groups Projects
Select Git revision
  • hpvm-release-epochs
  • main default
  • hpvm-release-epochs0
  • v2.0
  • v2.0rc
  • v1.0
  • v0.5
7 results
Compare
Name Last commit Last update
..
benchmarks
dnn_benchmarks
regressionTests
unitTests
.gitignore
CMakeLists.txt
README.md
lit.cfg.py
lit.site.cfg.py.in
README.md

HPVM Test and Benchmarks

Directory Organization

This directory is organized as follows:

  • unitTests/ and regressionTests/: unit and regression tests for HPVM. These are LLVM-bitcode test cases for HPVM passes.

  • benchmarks/: includes a few applications written in HPVM-C, a template, and directions for compiling and running these benchmarks.

  • dnn_benchmarks/: ten (10) DNN benchmarks in HPVM-C, Keras and PyTorch, supported by ApproxHPVM. This tests HPVM as well as the Keras and PyTorch frontends.

    • dnn_benchmarks/hpvm-c contains the HPVM-C version of these DNNs. Their organization and usage are similar to the benchmarks under benchmarks/.
    • dnn_benchmarks/keras contains these DNNs implemented in Keras, and code for generating them down to HPVM-C (testing Keras frontend).
    • dnn_benchmarks/pytorch contains these DNNs in PyTorch and code for generating them down to HPVM-C (testing PyTorch/ONNX frontend).

    The code generated from Keras and PyTorch frontend should be largely similar and functionally equivalent.

Running Test Cases and Benchmarks

The easiest way to run unitTests/ and regressionTests/ is to build the target hpvm-check in the global build directory: make -j hpvm-check. hpvm-check doesn't automatically run benchmarks/ and dnn_benchmarks as they are extremely time-consuming.

benchmarks/ can only be compiled in-source with make. We are working to migrate it into the cmake system.

HPVM-C DNN Benchmarks

To build all dnn_benchmarks/hpvm-c, use make -j dnn_benchmarks.

Alternatively, it's possible to build just 1 DNN benchmark. The output of CMake shows a list of these benchmarks as target names, starting with

List of test dnn benchmarks: alexnet2_cifar10;alexnet2_cifar10...

Currently, there are 20 of them. These are:
lenet_mnist lenet_mnist_cudnn
alexnet_cifar10 alexnet_cifar10_cudnn
alexnet2_cifar10 alexnet2_cifar10_cudnn
vgg16_cifar10 vgg16_cifar10_cudnn
vgg16_cifar100 vgg16_cifar100_cudnn
mobilenet_cifar10 mobilenet_cifar10_cudnn
resnet18_cifar10 resnet18_cifar10_cudnn
alexnet_imagenet alexnet_imagenet_cudnn
vgg16_imagenet vgg16_imagenet_cudnn
resnet50_imagenet resnet50_imagenet_cudnn

_cudnn suffix indicates the code is generated onto cuDNN functions. Otherwise they are generated to tensor_runtime DNN functions which are hand-written in CUDA.

DNN Frontends

TODO: figure out how to

  1. Auto run all hpvm-c DNN benchmarks
  2. Compare the output accuracy to groundtruth
  3. Auto run Keras and PyTorch tests (generating, compiling and running all DNNs)