From 377126bf3c81df0948f8344d0da6cd47c55bb6fa Mon Sep 17 00:00:00 2001
From: Hashim Sharif <hsharif3@miranda.cs.illinois.edu>
Date: Fri, 19 Mar 2021 01:33:04 -0500
Subject: [PATCH] Adding separate README for Keras benchmarks under
dnn_benchmarks
---
hpvm/test/dnn_benchmarks/keras/README.md | 137 +++++++++++++++++++++++
1 file changed, 137 insertions(+)
create mode 100644 hpvm/test/dnn_benchmarks/keras/README.md
diff --git a/hpvm/test/dnn_benchmarks/keras/README.md b/hpvm/test/dnn_benchmarks/keras/README.md
new file mode 100644
index 0000000000..f199de7d62
--- /dev/null
+++ b/hpvm/test/dnn_benchmarks/keras/README.md
@@ -0,0 +1,137 @@
+# Keras Frontend
+
+## Requirements
+
+* python >= 3.6
+* pip >= 9
+
+## Installing the Keras Frontend Package
+
+At the root of this project (`/projects/keras/`) install the Keras frontend pip package as:
+
+```
+pip install -e .
+```
+
+## Download CNN Model Files
+
+The weight (model) and data files to use with the CNN benchmarks are hosted on Git LFS and need to separately downloaded. This can be done using:
+
+```
+git lfs fetch
+git lfs checkout
+```
+
+**NOTE:** Data donwload is necesary before running benchmarks
+
+## Running Benchmaks
+
+List of benchmarks and the expected accuracies:
+
+| Benchmark | Accuracy |
+| ----------- | ----------- |
+| AlexNet-CIFAR10 | 79.28 |
+| AlexNet2-CIFAR10 | 84.98 |
+| AlexNet-ImageNet | 56.30 |
+| LeNet-MNIST | 98.70 |
+| MobileNet-CIFAR10 | 84.42 |
+| ResNet18-CIFAR10 | 89.56 |
+| ResNet50-ImageNet | 75.10 |
+| VGG16-CIFAR10 | 89.96 |
+| VGG16-CIFAR100 | 66.50 |
+| VGG16-ImageNet | 69.46 |
+
+
+### Synopsis
+
+```
+python ${BENCH_NAME}.py [hpvm_reload|keras_reload] [frontend] [compile]
+
+```
+
+
+**Command-line Parameters**
+
+`hpvm_reload` : Reloads HPVM weights ('.bin' binary format used in `model_params` found here: https://gitlab.engr.illinois.edu/llvm/hpvm/-/tree/approx_hpvm_reorg_keras/hpvm/test/dnn_benchmarks/model_params) from directory path specified in the `reload_dir` parameter set in code - this is described in "Parameters to Change in Code" (below).
+
+`keras_reload`: Alternatively, reload weights in Keras `.h5` file format with path to file specified in `keras_model_file` described in "Parameters to Change in Code" (below).
+
+`frontend`: Invokes the HPVM frontend and dumps weights (in HPVM `.bin` format) in the output directory specified. The parameters that control where data and source files are dumped are specified by parameters `data_dir` and `src_dir`, respectively. These are described below.
+
+`compile`: Optional Parameter. When specified, it compiles the HPVM-C code generated by the frontend into an HPVM binary under the directory specified by `src_dir` (described below). If `src_dir` path exists, a unique directory (which appends a unique ID) is created.
+The binary is built with the name `HPVM_binary`.
+
+**NOTE:** Before running `HPVM_binary` necessary to set CUDA and CUDNN paths with:
+
+```
+source ${PATH_TO_YOUR_HPVM_ROOT}/hpvm/set_paths.sh
+```
+
+**Parameters to Change in Code**
+
+The AlexNet source is commented with explanations on how to use the Keras frontend interface. AlexNet source is [here](https://gitlab.engr.illinois.edu/llvm/hpvm/-/blob/approx_hpvm_reorg_keras/hpvm/projects/keras/src/alexnet.py).
+
+* `NAME`: Benchmark Name - Can be set to any desired value
+
+* `reload_dir`: Path to directory from where to reload weights in HPVM format. This directory is used to reload weights if `hpvm_reload` command-line option is used.
+
+* `keras_model_file`: Path to Keras .h5 model file to reload weigths from. Either of `reload_dir` or `keras_model_file` can be used.
+`keras_model_file` is used when `keras_reload` commad-line parameter is used with the Benchmark script.
+
+* `data_dir`: Directory to dump weights specified specified in [constructor](https://gitlab.engr.illinois.edu/llvm/hpvm/-/blob/approx_hpvm_reorg_keras/hpvm/projects/keras/src/Benchmark.py#L21)
+
+* `src_dir`: Directory to dump ApproxHPVM sources in HPVM-C (C with HPVM compiler intrinsics) specified in [constructor](https://gitlab.engr.illinois.edu/llvm/hpvm/-/blob/approx_hpvm_reorg_keras/hpvm/projects/keras/src/Benchmark.py#L22)
+
+* `num_classes`: number of output classes - dependent on the dataset used. For CIFAR10, `num_classes` is 10, CIFAR100 has 100 classes,
+ for ImageNet, number of classes is 1000.
+
+* `batch_size`: This parameter controls the size of each batch that is processed in HPVM. The batch size should be kept as large as the GPU memory
+can support. This parameter should be adapted according to the memory size of the deployed device.
+
+
+
+### Using the Frontend with Custom (New) Benchmarks
+
+Any new benchmarks must inherit from the commom parent `Benchmark` class
+and override the virtual functions for building the model, training,
+and data preprocessing. These methods are described below:
+
+
+`def buildModel(self)`:
+Constructs and returns a keras model
+
+`def data_preprocess(self)`:
+returns X_train, y_train, X_test, y_test, X_tuner, and y_tuner data (in that order):
+These are described here:
+
+* `X_train:` Training data (fp32) in NCHW format
+* `y_train:` Training labels (int32)
+
+* `X_test:` Testing/Evaluation data in NCHW format
+* `y_test:` Testing/Evaluation labels
+
+* `X_tuner:` Data to be used for autotuning
+* `y_tuner:` Labels corresponding to tuning data
+
+
+`def trainModel(self, model, X_train, y_train, X_test, y_test)`:
+Trains the Keras model constructed in `buildModel` and is expected to return the
+trained keras model - training parameters should be tuned here.
+
+
+## Automated Tests
+
+`scripts/test_benchmarks.py` is an automated test script that evaluates the accuracy of each Benchmark in Keras and HPVM (after comilation using HPVM Compiler) and compares the accuracy of each binary to the known correct accuracy. Run from root of `/test/dnn_benchmarks/keras`:
+
+```
+python test_benchmarks.py
+```
+
+
+## Suppported Operations
+
+List of supported operations and limitations detailed in https://gitlab.engr.illinois.edu/llvm/hpvm/-/blob/approx_hpvm_reorg_keras/hpvm/projects/keras/docs/Support.md
+
+
+
+
--
GitLab