-
- Downloads
[SPARK-20642][CORE] Store FsHistoryProvider listing data in a KVStore.
The application listing is still generated from event logs, but is now stored in a KVStore instance. By default an in-memory store is used, but a new config allows setting a local disk path to store the data, in which case a LevelDB store will be created. The provider stores things internally using the public REST API types; I believe this is better going forward since it will make it easier to get rid of the internal history server API which is mostly redundant at this point. I also added a finalizer to LevelDBIterator, to make sure that resources are eventually released. This helps when code iterates but does not exhaust the iterator, thus not triggering the auto-close code. HistoryServerSuite was modified to not re-start the history server unnecessarily; this makes the json validation tests run more quickly. Author: Marcelo Vanzin <vanzin@cloudera.com> Closes #18887 from vanzin/SPARK-20642.
Showing
- common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDB.java 24 additions, 9 deletions.../src/main/java/org/apache/spark/util/kvstore/LevelDB.java
- common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java 10 additions, 0 deletions...n/java/org/apache/spark/util/kvstore/LevelDBIterator.java
- core/pom.xml 5 additions, 0 deletionscore/pom.xml
- core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala 359 additions, 300 deletions...a/org/apache/spark/deploy/history/FsHistoryProvider.scala
- core/src/main/scala/org/apache/spark/deploy/history/config.scala 48 additions, 0 deletions...c/main/scala/org/apache/spark/deploy/history/config.scala
- core/src/main/scala/org/apache/spark/status/api/v1/api.scala 12 additions, 3 deletionscore/src/main/scala/org/apache/spark/status/api/v1/api.scala
- core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala 58 additions, 13 deletions.../apache/spark/deploy/history/FsHistoryProviderSuite.scala
- core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala 10 additions, 7 deletions.../org/apache/spark/deploy/history/HistoryServerSuite.scala
- docs/monitoring.md 9 additions, 0 deletionsdocs/monitoring.md
- scalastyle-config.xml 1 addition, 1 deletionscalastyle-config.xml
Loading
Please register or sign in to comment