-
- Downloads
[SPARK-18280][CORE] Fix potential deadlock in `StandaloneSchedulerBackend.dead`
## What changes were proposed in this pull request? "StandaloneSchedulerBackend.dead" is called in a RPC thread, so it should not call "SparkContext.stop" in the same thread. "SparkContext.stop" will block until all RPC threads exit, if it's called inside a RPC thread, it will be dead-lock. This PR add a thread local flag inside RPC threads. `SparkContext.stop` uses it to decide if launching a new thread to stop the SparkContext. ## How was this patch tested? Jenkins Author: Shixiong Zhu <shixiong@databricks.com> Closes #15775 from zsxwing/SPARK-18280.
Showing
- core/src/main/scala/org/apache/spark/SparkContext.scala 20 additions, 2 deletionscore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/rpc/RpcEnv.scala 4 additions, 0 deletionscore/src/main/scala/org/apache/spark/rpc/RpcEnv.scala
- core/src/main/scala/org/apache/spark/rpc/netty/Dispatcher.scala 1 addition, 0 deletions...rc/main/scala/org/apache/spark/rpc/netty/Dispatcher.scala
- core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala 3 additions, 0 deletions...c/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala
- core/src/test/scala/org/apache/spark/rpc/RpcEnvSuite.scala 13 additions, 0 deletionscore/src/test/scala/org/apache/spark/rpc/RpcEnvSuite.scala
Loading
Please register or sign in to comment