Skip to content
Snippets Groups Projects
  • li-zhihui's avatar
    28dbae85
    [SPARK-2635] Fix race condition at SchedulerBackend.isReady in standalone mode · 28dbae85
    li-zhihui authored
    In SPARK-1946(PR #900), configuration <code>spark.scheduler.minRegisteredExecutorsRatio</code> was introduced. However, in standalone mode, there is a race condition where isReady() can return true because totalExpectedExecutors has not been correctly set.
    
    Because expected executors is uncertain in standalone mode, the PR try to use CPU cores(<code>--total-executor-cores</code>) as expected resources to judge whether SchedulerBackend is ready.
    
    Author: li-zhihui <zhihui.li@intel.com>
    Author: Li Zhihui <zhihui.li@intel.com>
    
    Closes #1525 from li-zhihui/fixre4s and squashes the following commits:
    
    e9a630b [Li Zhihui] Rename variable totalExecutors and clean codes
    abf4860 [Li Zhihui] Push down variable totalExpectedResources to children classes
    ca54bd9 [li-zhihui] Format log with String interpolation
    88c7dc6 [li-zhihui] Few codes and docs refactor
    41cf47e [li-zhihui] Fix race condition at SchedulerBackend.isReady in standalone mode
    28dbae85
    History
    [SPARK-2635] Fix race condition at SchedulerBackend.isReady in standalone mode
    li-zhihui authored
    In SPARK-1946(PR #900), configuration <code>spark.scheduler.minRegisteredExecutorsRatio</code> was introduced. However, in standalone mode, there is a race condition where isReady() can return true because totalExpectedExecutors has not been correctly set.
    
    Because expected executors is uncertain in standalone mode, the PR try to use CPU cores(<code>--total-executor-cores</code>) as expected resources to judge whether SchedulerBackend is ready.
    
    Author: li-zhihui <zhihui.li@intel.com>
    Author: Li Zhihui <zhihui.li@intel.com>
    
    Closes #1525 from li-zhihui/fixre4s and squashes the following commits:
    
    e9a630b [Li Zhihui] Rename variable totalExecutors and clean codes
    abf4860 [Li Zhihui] Push down variable totalExpectedResources to children classes
    ca54bd9 [li-zhihui] Format log with String interpolation
    88c7dc6 [li-zhihui] Few codes and docs refactor
    41cf47e [li-zhihui] Fix race condition at SchedulerBackend.isReady in standalone mode