Skip to content
Snippets Groups Projects
Commit 70d81466 authored by Tatiana Borisova's avatar Tatiana Borisova Committed by Josh Rosen
Browse files

[SPARK-3150] Fix NullPointerException in in Spark recovery: Add initializing...

[SPARK-3150] Fix NullPointerException in in Spark recovery: Add initializing default values in DriverInfo.init()

The issue happens when Spark is run standalone on a cluster.
When master and driver fall simultaneously on one node in a cluster, master tries to recover its state and restart spark driver.
While restarting driver, it falls with NPE exception (stacktrace is below).
After falling, it restarts and tries to recover its state and restart Spark driver again. It happens over and over in an infinite cycle.
Namely, Spark tries to read DriverInfo state from zookeeper, but after reading it happens to be null in DriverInfo.worker.

https://issues.apache.org/jira/browse/SPARK-3150

Author: Tatiana Borisova <tanyatik@yandex.ru>

Closes #2062 from tanyatik/spark-3150 and squashes the following commits:

9936043 [Tatiana Borisova] Add initializing default values in DriverInfo.init()
parent 76e3ba42
No related branches found
No related tags found
No related merge requests found
...@@ -33,4 +33,17 @@ private[spark] class DriverInfo( ...@@ -33,4 +33,17 @@ private[spark] class DriverInfo(
@transient var exception: Option[Exception] = None @transient var exception: Option[Exception] = None
/* Most recent worker assigned to this driver */ /* Most recent worker assigned to this driver */
@transient var worker: Option[WorkerInfo] = None @transient var worker: Option[WorkerInfo] = None
init()
private def readObject(in: java.io.ObjectInputStream): Unit = {
in.defaultReadObject()
init()
}
private def init(): Unit = {
state = DriverState.SUBMITTED
worker = None
exception = None
}
} }
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment