- Got rid of global SparkContext.globalConf
- Pass SparkConf to serializers and compression codecs
- Made SparkConf public instead of private[spark]
- Improved API of SparkContext and SparkConf
- Switched executor environment vars to be passed through SparkConf
- Fixed some places that were still using system properties
- Fixed some tests, though others are still failing

This still fails several tests in core, repl and streaming, likely due
to properties not being set or cleared correctly (some of the tests run
fine in isolation).

Incorporated Patrick's feedback comment on #211 and made maven build/dep-resolution atleast a bit faster.

With this scheduler, the user application is launched locally,
While the executor will be launched by YARN on remote nodes.

This enables spark-shell to run upon YARN.

Passed the sbt/sbt compile and test

Also remove unused imports as I found them along the way.
Remove return statements when returning value in the Scala code.

Passing compile and tests.

-) Remove some of unused imports as I found them
-) Remove ";" in the imports statements
-) Remove () at the end of method call like size that does not have size effect.

Allow spark on yarn to be run from HDFS. Allows the spark.jar, app.jar, and log4j.properties to be put into hdfs.

The code in LocalScheduler/LocalTaskSetManager was nearly identical
to the code in ClusterScheduler/ClusterTaskSetManager. The redundancy
made making updating the schedulers unnecessarily painful and error-
prone. This commit combines the two into a single TaskScheduler/
TaskSetManager.

about inclusion of spark.jar and app.jar

the classpaths

Support distributed cache files and archives on spark on yarn and attempt to cleanup the staging directory on exit

This uses a seperate Hadoop version for YARN artifact. This means when people link against
spark-yarn, things will resolve correctly.