Skip to content
Snippets Groups Projects
  • Dongjoon Hyun's avatar
    cdce4e62
    [SPARK-15031][EXAMPLE] Use SparkSession in Scala/Python/Java example. · cdce4e62
    Dongjoon Hyun authored
    ## What changes were proposed in this pull request?
    
    This PR aims to update Scala/Python/Java examples by replacing `SQLContext` with newly added `SparkSession`.
    
    - Use **SparkSession Builder Pattern** in 154(Scala 55, Java 52, Python 47) files.
    - Add `getConf` in Python SparkContext class: `python/pyspark/context.py`
    - Replace **SQLContext Singleton Pattern** with **SparkSession Singleton Pattern**:
      - `SqlNetworkWordCount.scala`
      - `JavaSqlNetworkWordCount.java`
      - `sql_network_wordcount.py`
    
    Now, `SQLContexts` are used only in R examples and the following two Python examples. The python examples are untouched in this PR since it already fails some unknown issue.
    - `simple_params_example.py`
    - `aft_survival_regression.py`
    
    ## How was this patch tested?
    
    Manual.
    
    Author: Dongjoon Hyun <dongjoon@apache.org>
    
    Closes #12809 from dongjoon-hyun/SPARK-15031.
    cdce4e62
    History
    [SPARK-15031][EXAMPLE] Use SparkSession in Scala/Python/Java example.
    Dongjoon Hyun authored
    ## What changes were proposed in this pull request?
    
    This PR aims to update Scala/Python/Java examples by replacing `SQLContext` with newly added `SparkSession`.
    
    - Use **SparkSession Builder Pattern** in 154(Scala 55, Java 52, Python 47) files.
    - Add `getConf` in Python SparkContext class: `python/pyspark/context.py`
    - Replace **SQLContext Singleton Pattern** with **SparkSession Singleton Pattern**:
      - `SqlNetworkWordCount.scala`
      - `JavaSqlNetworkWordCount.java`
      - `sql_network_wordcount.py`
    
    Now, `SQLContexts` are used only in R examples and the following two Python examples. The python examples are untouched in this PR since it already fails some unknown issue.
    - `simple_params_example.py`
    - `aft_survival_regression.py`
    
    ## How was this patch tested?
    
    Manual.
    
    Author: Dongjoon Hyun <dongjoon@apache.org>
    
    Closes #12809 from dongjoon-hyun/SPARK-15031.