-
- Downloads
[SPARK-20935][STREAMING] Always close WriteAheadLog and make it idempotent
## What changes were proposed in this pull request? This PR proposes to stop `ReceiverTracker` to close `WriteAheadLog` whenever it is and make `WriteAheadLog` and its implementations idempotent. ## How was this patch tested? Added a test in `WriteAheadLogSuite`. Note that the added test looks passing even if it closes twice (namely even without the changes in `FileBasedWriteAheadLog` and `BatchedWriteAheadLog`. It looks both are already idempotent but this is a rather sanity check. Author: hyukjinkwon <gurwls223@gmail.com> Closes #18224 from HyukjinKwon/streaming-closing.
Showing
- streaming/src/main/java/org/apache/spark/streaming/util/WriteAheadLog.java 1 addition, 1 deletion...n/java/org/apache/spark/streaming/util/WriteAheadLog.java
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverTracker.scala 10 additions, 17 deletions...rg/apache/spark/streaming/scheduler/ReceiverTracker.scala
- streaming/src/main/scala/org/apache/spark/streaming/util/BatchedWriteAheadLog.scala 6 additions, 7 deletions...rg/apache/spark/streaming/util/BatchedWriteAheadLog.scala
- streaming/src/main/scala/org/apache/spark/streaming/util/FileBasedWriteAheadLog.scala 5 additions, 3 deletions.../apache/spark/streaming/util/FileBasedWriteAheadLog.scala
- streaming/src/test/scala/org/apache/spark/streaming/scheduler/ReceiverTrackerSuite.scala 2 additions, 0 deletions...ache/spark/streaming/scheduler/ReceiverTrackerSuite.scala
- streaming/src/test/scala/org/apache/spark/streaming/util/WriteAheadLogSuite.scala 2 additions, 0 deletions.../org/apache/spark/streaming/util/WriteAheadLogSuite.scala
Loading
Please register or sign in to comment