Author: Neville Li <neville@spotify.com>

Closes #812 from nevillelyh/neville/v1.0 and squashes the following commits:

0dc33ed [Neville Li] Fix spark-submit path in pyspark
becec64 [Neville Li] Fix spark-submit path in spark-shell