Skip to content
Snippets Groups Projects
Commit c529dcea authored by Mark Hamstra's avatar Mark Hamstra
Browse files

Avoid a lump of coal (NPE) in JobProgressListener's stocking.

parent 56094bcd
No related branches found
No related tags found
No related merge requests found
...@@ -146,12 +146,9 @@ private[spark] class JobProgressListener(val sc: SparkContext) extends SparkList ...@@ -146,12 +146,9 @@ private[spark] class JobProgressListener(val sc: SparkContext) extends SparkList
// update duration // update duration
y.taskTime += taskEnd.taskInfo.duration y.taskTime += taskEnd.taskInfo.duration
taskEnd.taskMetrics.shuffleReadMetrics.foreach { shuffleRead => Option(taskEnd.taskMetrics).foreach { taskMetrics =>
y.shuffleRead += shuffleRead.remoteBytesRead taskMetrics.shuffleReadMetrics.foreach { y.shuffleRead += _.remoteBytesRead }
} taskMetrics.shuffleWriteMetrics.foreach { y.shuffleWrite += _.shuffleBytesWritten }
taskEnd.taskMetrics.shuffleWriteMetrics.foreach { shuffleWrite =>
y.shuffleWrite += shuffleWrite.shuffleBytesWritten
} }
} }
case _ => {} case _ => {}
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment