Skip to content
Snippets Groups Projects
  • jerryshao's avatar
    06c05441
    [SPARK-20981][SPARKSUBMIT] Add new configuration spark.jars.repositories as... · 06c05441
    jerryshao authored
    [SPARK-20981][SPARKSUBMIT] Add new configuration spark.jars.repositories as equivalence of --repositories
    
    ## What changes were proposed in this pull request?
    
    In our use case of launching Spark applications via REST APIs (Livy), there's no way for user to specify command line arguments, all Spark configurations are set through configurations map. For "--repositories" because there's no equivalent Spark configuration, so we cannot specify the custom repository through configuration.
    
    So here propose to add "--repositories" equivalent configuration in Spark.
    
    ## How was this patch tested?
    
    New UT added.
    
    Author: jerryshao <sshao@hortonworks.com>
    
    Closes #18201 from jerryshao/SPARK-20981.
    06c05441
    History
    [SPARK-20981][SPARKSUBMIT] Add new configuration spark.jars.repositories as...
    jerryshao authored
    [SPARK-20981][SPARKSUBMIT] Add new configuration spark.jars.repositories as equivalence of --repositories
    
    ## What changes were proposed in this pull request?
    
    In our use case of launching Spark applications via REST APIs (Livy), there's no way for user to specify command line arguments, all Spark configurations are set through configurations map. For "--repositories" because there's no equivalent Spark configuration, so we cannot specify the custom repository through configuration.
    
    So here propose to add "--repositories" equivalent configuration in Spark.
    
    ## How was this patch tested?
    
    New UT added.
    
    Author: jerryshao <sshao@hortonworks.com>
    
    Closes #18201 from jerryshao/SPARK-20981.