-
- Downloads
[SPARK-22123][CORE] Add latest failure reason for task set blacklist
## What changes were proposed in this pull request? This patch add latest failure reason for task set blacklist.Which can be showed on spark ui and let user know failure reason directly. Till now , every job which aborted by completed blacklist just show log like below which has no more information: `Aborting $taskSet because task $indexInTaskSet (partition $partition) cannot run anywhere due to node and executor blacklist. Blacklisting behavior cannot run anywhere due to node and executor blacklist.Blacklisting behavior can be configured via spark.blacklist.*."` **After modify:** ``` Aborting TaskSet 0.0 because task 0 (partition 0) cannot run anywhere due to node and executor blacklist. Most recent failure: Some(Lost task 0.1 in stage 0.0 (TID 3,xxx, executor 1): java.lang.Exception: Fake error! at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:73) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:305) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ). Blacklisting behavior can be configured via spark.blacklist.*. ``` ## How was this patch tested? Unit test and manually test. Author: zhoukang <zhoukang199191@gmail.com> Closes #19338 from caneGuy/zhoukang/improve-blacklist.
Showing
- core/src/main/scala/org/apache/spark/scheduler/TaskSetBlacklist.scala 13 additions, 1 deletion...n/scala/org/apache/spark/scheduler/TaskSetBlacklist.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala 10 additions, 5 deletions...ain/scala/org/apache/spark/scheduler/TaskSetManager.scala
- core/src/test/scala/org/apache/spark/scheduler/BlacklistIntegrationSuite.scala 3 additions, 2 deletions...rg/apache/spark/scheduler/BlacklistIntegrationSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala 39 additions, 21 deletions...la/org/apache/spark/scheduler/BlacklistTrackerSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala 8 additions, 3 deletions...a/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/TaskSetBlacklistSuite.scala 30 additions, 15 deletions...la/org/apache/spark/scheduler/TaskSetBlacklistSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala 1 addition, 1 deletion...cala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
Loading
Please register or sign in to comment