-
- Downloads
Merge pull request #309 from mateiz/conf2
SPARK-544. Migrate configuration to a SparkConf class This is still a work in progress based on Prashant and Evan's code. So far I've done the following: - Got rid of global SparkContext.globalConf - Passed SparkConf to serializers and compression codecs - Made SparkConf public instead of private[spark] - Improved API of SparkContext and SparkConf - Switched executor environment vars to be passed through SparkConf - Fixed some places that were still using system properties - Fixed some tests, though others are still failing This still fails several tests in core, repl and streaming, likely due to properties not being set or cleared correctly (some of the tests run fine in isolation). But the API at least is hopefully ready for review. Unfortunately there was a lot of global stuff before due to a "SparkContext.globalConf" method that let you set a "default" configuration of sorts, which meant I had to make some pretty big changes.
No related branches found
No related tags found
Showing
- core/src/main/scala/org/apache/spark/Accumulators.scala 4 additions, 4 deletionscore/src/main/scala/org/apache/spark/Accumulators.scala
- core/src/main/scala/org/apache/spark/MapOutputTracker.scala 6 additions, 5 deletionscore/src/main/scala/org/apache/spark/MapOutputTracker.scala
- core/src/main/scala/org/apache/spark/Partitioner.scala 2 additions, 2 deletionscore/src/main/scala/org/apache/spark/Partitioner.scala
- core/src/main/scala/org/apache/spark/SparkConf.scala 189 additions, 0 deletionscore/src/main/scala/org/apache/spark/SparkConf.scala
- core/src/main/scala/org/apache/spark/SparkContext.scala 146 additions, 95 deletionscore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/SparkEnv.scala 30 additions, 24 deletionscore/src/main/scala/org/apache/spark/SparkEnv.scala
- core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala 20 additions, 1 deletion...in/scala/org/apache/spark/api/java/JavaSparkContext.scala
- core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala 2 additions, 2 deletions...rc/main/scala/org/apache/spark/api/python/PythonRDD.scala
- core/src/main/scala/org/apache/spark/broadcast/Broadcast.scala 4 additions, 4 deletions...src/main/scala/org/apache/spark/broadcast/Broadcast.scala
- core/src/main/scala/org/apache/spark/broadcast/BroadcastFactory.scala 3 additions, 1 deletion...n/scala/org/apache/spark/broadcast/BroadcastFactory.scala
- core/src/main/scala/org/apache/spark/broadcast/HttpBroadcast.scala 25 additions, 18 deletions...main/scala/org/apache/spark/broadcast/HttpBroadcast.scala
- core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala 23 additions, 22 deletions...n/scala/org/apache/spark/broadcast/TorrentBroadcast.scala
- core/src/main/scala/org/apache/spark/deploy/FaultToleranceTest.scala 2 additions, 2 deletions...in/scala/org/apache/spark/deploy/FaultToleranceTest.scala
- core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala 4 additions, 3 deletions...ain/scala/org/apache/spark/deploy/LocalSparkCluster.scala
- core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala 9 additions, 5 deletions.../main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala
- core/src/main/scala/org/apache/spark/deploy/client/Client.scala 6 additions, 7 deletions...rc/main/scala/org/apache/spark/deploy/client/Client.scala
- core/src/main/scala/org/apache/spark/deploy/client/TestClient.scala 6 additions, 4 deletions...ain/scala/org/apache/spark/deploy/client/TestClient.scala
- core/src/main/scala/org/apache/spark/deploy/master/Master.scala 23 additions, 18 deletions...rc/main/scala/org/apache/spark/deploy/master/Master.scala
- core/src/main/scala/org/apache/spark/deploy/master/MasterArguments.scala 6 additions, 5 deletions...cala/org/apache/spark/deploy/master/MasterArguments.scala
- core/src/main/scala/org/apache/spark/deploy/master/SparkZooKeeperSession.scala 4 additions, 3 deletions...rg/apache/spark/deploy/master/SparkZooKeeperSession.scala
Loading
Please register or sign in to comment