-
- Downloads
[SPARK-8617][WEBUI] HistoryServer: Include in-progress files during cleanup
## What changes were proposed in this pull request? - Removed the`attempt.completed ` filter so cleaner would include the orphan inprogress files. - Use loading time for inprogress files as lastUpdated. Keep using the modTime for completed files. First one will prevent deletion of inprogress job files. Second one will ensure that lastUpdated time won't change for completed jobs in an event of HistoryServer reboot. ## How was this patch tested? Added new unittests and via existing tests. Author: Ergin Seyfe <eseyfe@fb.com> Closes #16165 from seyfe/clear_old_inprogress_files.
Showing
- core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala 7 additions, 3 deletions...a/org/apache/spark/deploy/history/FsHistoryProvider.scala
- core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala 48 additions, 2 deletions.../apache/spark/deploy/history/FsHistoryProviderSuite.scala
Loading
Please register or sign in to comment