-
- Downloads
[SPARK-2635] Fix race condition at SchedulerBackend.isReady in standalone mode
In SPARK-1946(PR #900), configuration <code>spark.scheduler.minRegisteredExecutorsRatio</code> was introduced. However, in standalone mode, there is a race condition where isReady() can return true because totalExpectedExecutors has not been correctly set. Because expected executors is uncertain in standalone mode, the PR try to use CPU cores(<code>--total-executor-cores</code>) as expected resources to judge whether SchedulerBackend is ready. Author: li-zhihui <zhihui.li@intel.com> Author: Li Zhihui <zhihui.li@intel.com> Closes #1525 from li-zhihui/fixre4s and squashes the following commits: e9a630b [Li Zhihui] Rename variable totalExecutors and clean codes abf4860 [Li Zhihui] Push down variable totalExpectedResources to children classes ca54bd9 [li-zhihui] Format log with String interpolation 88c7dc6 [li-zhihui] Few codes and docs refactor 41cf47e [li-zhihui] Fix race condition at SchedulerBackend.isReady in standalone mode
Showing
- core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala 14 additions, 16 deletions...ark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala 5 additions, 1 deletion...spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
- docs/configuration.md 7 additions, 6 deletionsdocs/configuration.md
- yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala 6 additions, 3 deletions.../spark/scheduler/cluster/YarnClientSchedulerBackend.scala
- yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClusterSchedulerBackend.scala 11 additions, 6 deletions...spark/scheduler/cluster/YarnClusterSchedulerBackend.scala
Loading
Please register or sign in to comment