Skip to content
Snippets Groups Projects
  • Bryan Cutler's avatar
    021c1970
    [SPARK-15456][PYSPARK] Fixed PySpark shell context initialization when HiveConf not present · 021c1970
    Bryan Cutler authored
    ## What changes were proposed in this pull request?
    
    When PySpark shell cannot find HiveConf, it will fallback to create a SparkSession from a SparkContext.  This fixes a bug caused by using a variable to SparkContext before it was initialized.
    
    ## How was this patch tested?
    
    Manually starting PySpark shell and using the SparkContext
    
    Author: Bryan Cutler <cutlerb@gmail.com>
    
    Closes #13237 from BryanCutler/pyspark-shell-session-context-SPARK-15456.
    021c1970
    History
    [SPARK-15456][PYSPARK] Fixed PySpark shell context initialization when HiveConf not present
    Bryan Cutler authored
    ## What changes were proposed in this pull request?
    
    When PySpark shell cannot find HiveConf, it will fallback to create a SparkSession from a SparkContext.  This fixes a bug caused by using a variable to SparkContext before it was initialized.
    
    ## How was this patch tested?
    
    Manually starting PySpark shell and using the SparkContext
    
    Author: Bryan Cutler <cutlerb@gmail.com>
    
    Closes #13237 from BryanCutler/pyspark-shell-session-context-SPARK-15456.