Skip to content
Snippets Groups Projects
Commit 05ab2948 authored by Yonathan Randolph's avatar Yonathan Randolph Committed by Sean Owen
Browse files

[SPARK-13906] Ensure that there are at least 2 dispatcher threads.

## What changes were proposed in this pull request?

Force at least two dispatcher-event-loop threads. Since SparkDeploySchedulerBackend (in AppClient) calls askWithRetry to CoarseGrainedScheduler in the same process, there the driver needs at least two dispatcher threads to prevent the dispatcher thread from hanging.

## How was this patch tested?

Manual.

Author: Yonathan Randolph <yonathangmail.com>

Author: Yonathan Randolph <yonathan@liftigniter.com>

Closes #11728 from yonran/SPARK-13906.
parent 431a3d04
No related branches found
No related tags found
No related merge requests found
......@@ -192,7 +192,7 @@ private[netty] class Dispatcher(nettyEnv: NettyRpcEnv) extends Logging {
/** Thread pool used for dispatching messages. */
private val threadpool: ThreadPoolExecutor = {
val numThreads = nettyEnv.conf.getInt("spark.rpc.netty.dispatcher.numThreads",
Runtime.getRuntime.availableProcessors())
math.max(2, Runtime.getRuntime.availableProcessors()))
val pool = ThreadUtils.newDaemonFixedThreadPool(numThreads, "dispatcher-event-loop")
for (i <- 0 until numThreads) {
pool.execute(new MessageLoop)
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment