Final fixes to all readmes

README.md

@@ -14,6 +14,8 @@ benchmark suite to test the compiler infrastructure.
 
 [HPVM-C Language Specification](/hpvm/docs/hpvm-c.md)
 
+[HPVM Compilation Process](/hpvm/docs/compilation.md)
+
 ## Dependencies
 The following components are required to be installed on your machine to build HPVM.
 
@@ -63,14 +65,14 @@ In order to manually build and install HPVM, GNU Make can be run using the follo
 make -j<number of threads>
 make install
 ```
-With all the aforementioned steps, HPVM should be built, installed and ready for use!
+With all the aforementioned steps, HPVM should be built, installed and ready for use.
 
 ## Benchmarks and Tests
 We are providing the following HPVM benchmarks:
-* Select benchmarks from the [Parboil](http://impact.crhc.illinois.edu/parboil/parboil.aspx) benchmark suite, located under [test/parboil](/hpvm/test/parboil).
-* An edge detection pipeline benchmark, located under [test/pipeline](/hpvm/test/pipeline).
-* A Camera ISP pipeline, located under [test/cava](/hpvm/test/pipeline), adapted from C code provided from our collaborators at [Harvard](http://vlsiarch.eecs.harvard.edu).
+* Select benchmarks from the [Parboil](http://impact.crhc.illinois.edu/parboil/parboil.aspx) benchmark suite, located under [test/parboil](/hpvm/test/benchmarks/parboil).
+* An edge detection pipeline benchmark, located under [test/pipeline](/hpvm/test/benchmarks/pipeline).
+* A Camera ISP pipeline, located under [test/cava](/hpvm/test/benchmarks/hpvm-cava), adapted from C code provided from our collaborators at [Harvard](http://vlsiarch.eecs.harvard.edu).
 
 We are also providing [unit tests](/hpvm/test/unitTests) and [regression tests](/hpvm/test/regressionTests).
 
-Benchmark descriptions and instructions on how to compile and run them are [here](/hpvm/test).
+Benchmark descriptions and instructions on how to compile and run them are [here](/hpvm/test/benchmarks).

hpvm/docs/compilation.md

+# HPVM Compilation Process
+Compilation of an HPVM program involves the following steps:
+
+1. `clang` takes an HPVM-C/C++ program (e.g. `main.c`) and produces an LLVM IR (`main.ll`) file that contains the HPVM-C function calls. The declarations of these functions are defined in `test/benchmark/include/hpvm.h`, which must be included in the program.
+2. `opt` takes (`main.ll`) and invoke the GenHPVM pass on it, which converts the HPVM-C function calls to HPVM intrinsics. This generates the HPVM textual representation (`main.hpvm.ll`).
+3. `opt` takes the HPVM textual representation (`main.hpvm.ll`) and invokes the following passes in sequence: 
+  * BuildDFG: Converts the textual representation to the internal HPVM representation.
+  * LocalMem and DFG2LLVM_NVPTX: Invoked only when GPU target is selected. Generates the kernel module (`main.kernels.ll`) and the portion of the host code that invokes the kernel into the host module (`main.host.ll`).
+  * DFG2LLVM_X86: Generates either all, or the remainder of the host module (`main.host.ll`) depending on the chosen target.
+  * ClearDFG: Deletes the internal HPVM representation from memory.
+4. `clang` is used to to compile any remaining project files that would be later linked with the host module.
+5. `llvm-link` takes the host module and all the other generate `ll` files, and links them with the HPVM runtime module (`hpvm-rt.bc`), to generate the linked host module (`main.host.linked.ll`). 
+6. Generate the executable code from the generated `ll` files for all parts of the program:
+  * GPU target: `llvm-cbe` takes the kernel module (`main.kernels.ll`) and generates an OpenCL representation of the kernels that will be invoked by the host.
+  * X86 target: `clang` takes the linked  host module (`main.host.linked.ll`) and generates the X86 binary.
\ No newline at end of file

hpvm/test/README.md

-# Using HPVM
-The below benchmarks are provided with HPVM, along with a template Makefile for user projects. In order to be able to build the existing benchmarks, a new `Makefile.config` must be created in [include](/hpvm/test/include) based on the existing `Makefile.config.example`. This configuration file must set up the following paths:
-* LLVM_BUILD_DIR: should point to your local `build` directory of HPVM.
-* CUDA_PATH: should point to your local CUDA installation.
-
-## Parboil
-Instructions to compile and run Parboil are provided in the following [README](/hpvm/test/parboil).
-
-## Harvard Camera Pipeline (HPVM-CAVA)
-Instructions to compile and run HPVM-CAVA are provided in the following [README](/hpvm/test/hpvm-cava).
-
-## Edge Detection Pipeline
-Instructions to compile and run Pipeline are provided in the following [README](/hpvm/test/pipeline).
-
-## Your own project
-See `template/` for an example Makefile and config.
-Include `hpvm.h` to use HPVM C api functions, found in the `test/include/hpvm.h`.
+#Test Directory Organization
+The test directory is organized as follows:
+* unitTests: Includes unit tests for HPVM.
+* regressionTests: Includes regression tests for HPVM.
+* benchmarks: Includes the current benchmarks and a template, as well as directions for compiling and running HPVM benchmarks.

hpvm/test/benchmarks/hpvm-cava/README.md

@@ -5,8 +5,6 @@ Harvard Camera Pipeline with HPVM intrinsics
 
 The camera pipeline is a simple five-stage image signal processor (ISP) which processes raw images (i.e., sensor inputs) into an image that can feed into a vision backend (e.g., a CNN).
 
-See the original camera/vision pipeline repo (repo: `yaoyuannnn/cava`) for details on each stage.
-
 ## How to Build and Test
 
 After building HPVM, the following steps are required to build and run the camera pipeline:
@@ -15,6 +13,6 @@ After building HPVM, the following steps are required to build and run the camer
 2. Run with `./cava-hpvm-<Target> example-tulip-small/raw_tulip-small.bin example-tulip-small/tulip-small`. 
     * `<Target>` can be either `seq` or `gpu` depending on what target is used to build.
     * This processes the raw image `example-tulip-small/raw_tulip-small.bin`. Note that raw images are different from bitmaps, so you might need to obtain them using special software.
-    * This generates: `tulip-small.bin` and `tulip-small-<stage>.bin` where `<stage>` represents the stage of the pipeline.
-3. Convert the binary outputs to a PNG with `./convert.sh example-tulip-small`
-4. View the resulting PNG at `example-tulip-small/tulip-small.png`. (As well as all the intermediary images for each stage `tulip-small-<stage>.png`)
+    * This generates: `tulip-small.bin` and `tulip-small-<stage>.bin` where `<stage>` represents the stage of the pipeline, in the directory `example-tulip-small`.
+3. Convert the binary outputs to a PNG with `./convert.sh example-tulip-small`.
+4. View the resulting PNG at `example-tulip-small/tulip-small.png`. (As well as all the intermediary images for each stage `tulip-small-<stage>.png`).

hpvm/test/benchmarks/include/Makefile.config.example

@@ -6,13 +6,13 @@ OPENCL_LIB_PATH=$(OPENCL_PATH)/lib64
 
 # This path should be set to your HPVM build directory
 LLVM_BUILD_DIR=PATH/TO/LOCAL/HPVM/BUILD
-HPVM_TEST_DIR=PATH/TO/HPVM/TEST/DIR
+HPVM_BENCH_DIR=PATH/TO/HPVM/BENCH/DIR
 
 CC = $(LLVM_BUILD_DIR)/bin/clang
-PLATFORM_CFLAGS = -I$(OPENCL_PATH)/include/CL/ -I$(HPVM_TEST_DIR)/include
+PLATFORM_CFLAGS = -I$(OPENCL_PATH)/include/CL/ -I$(HPVM_BENCH_DIR)/include
 
 CXX = $(LLVM_BUILD_DIR)/bin/clang++
-PLATFORM_CXXFLAGS = -I$(OPENCL_PATH)/include/CL/ -I$(HPVM_TEST_DIR)/include
+PLATFORM_CXXFLAGS = -I$(OPENCL_PATH)/include/CL/ -I$(HPVM_BENCH_DIR)/include
 
 LINKER = $(LLVM_BUILD_DIR)/bin/clang++
 PLATFORM_LDFLAGS = -lm -lpthread -lrt -lOpenCL -L$(OPENCL_LIB_PATH)

hpvm/test/benchmarks/template/README.md

-Compilation of a project with HPVM is a multi-step process.
-Let's look at the compilation of the `pipeline` test for gpu as an example.
-
-`clang` is used to produce an LLVM IR file that contains the HPVM intrinsics in the form of function calls.
-```
-/.../hpvm/build/bin/clang -Isrc/ -I -I/.../hpvm/llvm/include -I../include -I/.../hpvm/build/include -ffast-math -O3 -fno-lax-vector-conversions -fno-vectorize -fno-slp-vectorize -I/.../hpvm/llvm/include -I../include -I/.../hpvm/build/include -DDEVICE=GPU_TARGET -emit-llvm -S -o build/main.ll src/main.cc
-```
-
-`opt` is used to invoke the GenHPVM pass, which converts the HPVM function calls to LLVM intrinsics.
-```
-/.../hpvm/build/bin/opt -debug-only=genhpvm -load LLVMGenHPVM.so -genhpvm -globaldce -hpvm-timers-gen build/main.ll -S -o build/main.hpvm.ll
-```
-
-`opt` is used again to invoke the BuildDFG pass, which converts the textual representation to the internal HPVM representation.
-```
-/.../hpvm/build/bin/opt -debug -load LLVMBuildDFG.so -load LLVMLocalMem.so -load LLVMDFG2LLVM_NVPTX.so -load LLVMDFG2LLVM_X86.so -load LLVMClearDFG.so -localmem -dfg2llvm-nvptx -dfg2llvm-x86 -clearDFG -hpvm-timers-x86 -hpvm-timers-ptx -S build/main.hpvm.ll -o build/pipeline-gpu.host.ll
-```
-
-`llvm-cbe` is a C backend for LLVM. It is used here to create the OpenCL kernel.
-```
-/.../hpvm/build/bin/llvm-cbe -debug build/gpu/main.hpvm.ll.kernels.ll -o build/gpu/main.hpvm.ll.kernels.cl
-```
-
-`clang` is used again to compile a separate source file that contains I/O code.
-```
-/.../hpvm/build/bin/clang -Isrc/ -I -I/.../hpvm/llvm/include -I../include -I/.../hpvm/build/include -ffast-math -O3 -fno-lax-vector-conversions -fno-vectorize -fno-slp-vectorize -I/.../hpvm/llvm/include -I../include -I/.../hpvm/build/include -emit-llvm -S -o build/gpu/io.ll src/io.cc
-```
-
-`llvm-link` is used to link against the HPVM runtime.
-```
-/.../hpvm/build/bin/llvm-link build/gpu/pipeline-gpu.host.ll build/gpu/io.ll /.../hpvm/llvm/tools/hpvm/projects/hpvm-rt/hpvm-rt.ll -S -o build/gpu/pipeline-gpu.linked.ll
-```
-
-`clang++` is used to do the final linking against OpenCL and emit the binary.
-```
-/.../hpvm/build/bin/clang++ -O3 `pkg-config opencv --libs` -lm -lpthread -lrt -lOpenCL -L/software/cuda-9.1/lib64 build/gpu/pipeline-gpu.linked.ll -o pipeline-gpu
-```
\ No newline at end of file