-
- Downloads
[SPARK-1946] Submit tasks after (configured ratio) executors have been registered
Because submitting tasks and registering executors are asynchronous, in most situation, early stages' tasks run without preferred locality. A simple solution is sleeping few seconds in application, so that executors have enough time to register. The PR add 2 configuration properties to make TaskScheduler submit tasks after a few of executors have been registered. \# Submit tasks only after (registered executors / total executors) arrived the ratio, default value is 0 spark.scheduler.minRegisteredExecutorsRatio = 0.8 \# Whatever minRegisteredExecutorsRatio is arrived, submit tasks after the maxRegisteredWaitingTime(millisecond), default value is 30000 spark.scheduler.maxRegisteredExecutorsWaitingTime = 5000 Author: li-zhihui <zhihui.li@intel.com> Closes #900 from li-zhihui/master and squashes the following commits: b9f8326 [li-zhihui] Add logs & edit docs 1ac08b1 [li-zhihui] Add new configs to user docs 22ead12 [li-zhihui] Move waitBackendReady to postStartHook c6f0522 [li-zhihui] Bug fix: numExecutors wasn't set & use constant DEFAULT_NUMBER_EXECUTORS 4d6d847 [li-zhihui] Move waitBackendReady to TaskSchedulerImpl.start & some code refactor 0ecee9a [li-zhihui] Move waitBackendReady from DAGScheduler.submitStage to TaskSchedulerImpl.submitTasks 4261454 [li-zhihui] Add docs for new configs & code style ce0868a [li-zhihui] Code style, rename configuration property name of minRegisteredRatio & maxRegisteredWaitingTime 6cfb9ec [li-zhihui] Code style, revert default minRegisteredRatio of yarn to 0, driver get --num-executors in yarn/alpha 812c33c [li-zhihui] Fix driver lost --num-executors option in yarn-cluster mode e7b6272 [li-zhihui] support yarn-cluster 37f7dc2 [li-zhihui] support yarn mode(percentage style) 3f8c941 [li-zhihui] submit stage after (configured ratio of) executors have been registered
Showing
- core/src/main/scala/org/apache/spark/SparkContext.scala 10 additions, 1 deletioncore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/scheduler/SchedulerBackend.scala 1 addition, 0 deletions...n/scala/org/apache/spark/scheduler/SchedulerBackend.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala 15 additions, 0 deletions.../scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala 29 additions, 0 deletions...ark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala 1 addition, 0 deletions...spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
- docs/configuration.md 19 additions, 0 deletionsdocs/configuration.md
- yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala 1 addition, 0 deletions...cala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
- yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMasterArguments.scala 5 additions, 1 deletion...apache/spark/deploy/yarn/ApplicationMasterArguments.scala
- yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientClusterScheduler.scala 2 additions, 0 deletions.../spark/scheduler/cluster/YarnClientClusterScheduler.scala
- yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala 1 addition, 0 deletions.../spark/scheduler/cluster/YarnClientSchedulerBackend.scala
- yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClusterScheduler.scala 2 additions, 0 deletions...apache/spark/scheduler/cluster/YarnClusterScheduler.scala
- yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClusterSchedulerBackend.scala 40 additions, 0 deletions...spark/scheduler/cluster/YarnClusterSchedulerBackend.scala
- yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala 1 addition, 0 deletions...cala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
Loading
Please register or sign in to comment