Skip to content
  • Marcelo Vanzin's avatar
    74daf622
    [SPARK-20642][CORE] Store FsHistoryProvider listing data in a KVStore. · 74daf622
    Marcelo Vanzin authored
    The application listing is still generated from event logs, but is now stored
    in a KVStore instance. By default an in-memory store is used, but a new config
    allows setting a local disk path to store the data, in which case a LevelDB
    store will be created.
    
    The provider stores things internally using the public REST API types; I believe
    this is better going forward since it will make it easier to get rid of the
    internal history server API which is mostly redundant at this point.
    
    I also added a finalizer to LevelDBIterator, to make sure that resources are
    eventually released. This helps when code iterates but does not exhaust the
    iterator, thus not triggering the auto-close code.
    
    HistoryServerSuite was modified to not re-start the history server unnecessarily;
    this makes the json validation tests run more quickly.
    
    Author: Marcelo Vanzin <vanzin@cloudera.com>
    
    Closes #18887 from vanzin/SPARK-20642.
    74daf622
    [SPARK-20642][CORE] Store FsHistoryProvider listing data in a KVStore.
    Marcelo Vanzin authored
    The application listing is still generated from event logs, but is now stored
    in a KVStore instance. By default an in-memory store is used, but a new config
    allows setting a local disk path to store the data, in which case a LevelDB
    store will be created.
    
    The provider stores things internally using the public REST API types; I believe
    this is better going forward since it will make it easier to get rid of the
    internal history server API which is mostly redundant at this point.
    
    I also added a finalizer to LevelDBIterator, to make sure that resources are
    eventually released. This helps when code iterates but does not exhaust the
    iterator, thus not triggering the auto-close code.
    
    HistoryServerSuite was modified to not re-start the history server unnecessarily;
    this makes the json validation tests run more quickly.
    
    Author: Marcelo Vanzin <vanzin@cloudera.com>
    
    Closes #18887 from vanzin/SPARK-20642.
Loading