-
- Downloads
[SPARK-2260] Fix standalone-cluster mode, which was broken
The main thing was that spark configs were not propagated to the driver, and so applications that do not specify `master` or `appName` automatically failed. This PR fixes that and a couple of miscellaneous things that are related. One thing that may or may not be an issue is that the jars must be available on the driver node. In `standalone-cluster` mode, this effectively means these jars must be available on all the worker machines, since the driver is launched on one of them. The semantics here are not the same as `yarn-cluster` mode, where all the relevant jars are uploaded to a distributed cache automatically and shipped to the containers. This is probably not a concern, but still worth a mention. Author: Andrew Or <andrewor14@gmail.com> Closes #1538 from andrewor14/standalone-cluster and squashes the following commits: 8c11a0d [Andrew Or] Clean up imports / comments (minor) 2678d13 [Andrew Or] Handle extraJavaOpts properly 7660547 [Andrew Or] Merge branch 'master' of github.com:apache/spark into standalone-cluster 6f64a9b [Andrew Or] Revert changes in YARN 2f2908b [Andrew Or] Fix tests ed01491 [Andrew Or] Don't go overboard with escaping 8e105e1 [Andrew Or] Merge branch 'master' of github.com:apache/spark into standalone-cluster b890949 [Andrew Or] Abstract usages of converting spark opts to java opts 79f63a3 [Andrew Or] Move sparkProps into javaOpts 78752f8 [Andrew Or] Fix tests 5a9c6c7 [Andrew Or] Fix line too long c141a00 [Andrew Or] Don't display "unknown app" on driver log pages d7e2728 [Andrew Or] Avoid deprecation warning in standalone Client 6ceb14f [Andrew Or] Allow relevant configs to propagate to standalone Driver 7f854bc [Andrew Or] Fix test 855256e [Andrew Or] Fix standalone-cluster mode fd9da51 [Andrew Or] Formatting changes (minor)
Showing
- core/src/main/scala/org/apache/spark/SparkConf.scala 21 additions, 1 deletioncore/src/main/scala/org/apache/spark/SparkConf.scala
- core/src/main/scala/org/apache/spark/deploy/Client.scala 11 additions, 10 deletionscore/src/main/scala/org/apache/spark/deploy/Client.scala
- core/src/main/scala/org/apache/spark/deploy/Command.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/deploy/Command.scala
- core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala 5 additions, 7 deletions.../src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
- core/src/main/scala/org/apache/spark/deploy/client/TestClient.scala 3 additions, 3 deletions...ain/scala/org/apache/spark/deploy/client/TestClient.scala
- core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala 3 additions, 4 deletions...n/scala/org/apache/spark/deploy/worker/CommandUtils.scala
- core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala 2 additions, 1 deletion...n/scala/org/apache/spark/deploy/worker/DriverRunner.scala
- core/src/main/scala/org/apache/spark/deploy/worker/ExecutorRunner.scala 9 additions, 5 deletions...scala/org/apache/spark/deploy/worker/ExecutorRunner.scala
- core/src/main/scala/org/apache/spark/deploy/worker/ui/LogPage.scala 5 additions, 6 deletions...ain/scala/org/apache/spark/deploy/worker/ui/LogPage.scala
- core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala 7 additions, 2 deletions.../apache/spark/executor/CoarseGrainedExecutorBackend.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala 7 additions, 4 deletions...spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
- core/src/main/scala/org/apache/spark/util/Utils.scala 9 additions, 0 deletionscore/src/main/scala/org/apache/spark/util/Utils.scala
- core/src/test/scala/org/apache/spark/deploy/JsonProtocolSuite.scala 3 additions, 3 deletions...est/scala/org/apache/spark/deploy/JsonProtocolSuite.scala
- core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala 5 additions, 2 deletions...test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala
- core/src/test/scala/org/apache/spark/deploy/worker/DriverRunnerTest.scala 1 addition, 1 deletion...ala/org/apache/spark/deploy/worker/DriverRunnerTest.scala
- core/src/test/scala/org/apache/spark/deploy/worker/ExecutorRunnerTest.scala 1 addition, 1 deletion...a/org/apache/spark/deploy/worker/ExecutorRunnerTest.scala
Loading
Please register or sign in to comment