Skip to content
Snippets Groups Projects
Commit 8239fdcb authored by Eric Liang's avatar Eric Liang Committed by Andrew Or
Browse files

[SPARK-15520][SQL] SparkSession builder in python should also allow overriding...

[SPARK-15520][SQL] SparkSession builder in python should also allow overriding confs of existing sessions

## What changes were proposed in this pull request?

This fixes the python SparkSession builder to allow setting confs correctly. This was a leftover TODO from https://github.com/apache/spark/pull/13200.

## How was this patch tested?

Python doc tests.

cc andrewor14

Author: Eric Liang <ekl@databricks.com>

Closes #13289 from ericl/spark-15520.
parent 01e7b9c8
No related branches found
No related tags found
No related merge requests found
...@@ -138,24 +138,37 @@ class SparkSession(object): ...@@ -138,24 +138,37 @@ class SparkSession(object):
"""Gets an existing :class:`SparkSession` or, if there is no existing one, creates a """Gets an existing :class:`SparkSession` or, if there is no existing one, creates a
new one based on the options set in this builder. new one based on the options set in this builder.
This method first checks whether there is a valid thread-local SparkSession, This method first checks whether there is a valid global default SparkSession, and if
and if yes, return that one. It then checks whether there is a valid global yes, return that one. If no valid global default SparkSession exists, the method
default SparkSession, and if yes, return that one. If no valid global default creates a new SparkSession and assigns the newly created SparkSession as the global
SparkSession exists, the method creates a new SparkSession and assigns the default.
newly created SparkSession as the global default.
>>> s1 = SparkSession.builder.config("k1", "v1").getOrCreate()
>>> s1.conf.get("k1") == "v1"
True
In case an existing SparkSession is returned, the config options specified In case an existing SparkSession is returned, the config options specified
in this builder will be applied to the existing SparkSession. in this builder will be applied to the existing SparkSession.
>>> s2 = SparkSession.builder.config("k2", "v2").getOrCreate()
>>> s1.conf.get("k1") == s2.conf.get("k1")
True
>>> s1.conf.get("k2") == s2.conf.get("k2")
True
""" """
with self._lock: with self._lock:
from pyspark.conf import SparkConf
from pyspark.context import SparkContext from pyspark.context import SparkContext
from pyspark.sql.context import SQLContext from pyspark.conf import SparkConf
sparkConf = SparkConf() session = SparkSession._instantiatedContext
if session is None:
sparkConf = SparkConf()
for key, value in self._options.items():
sparkConf.set(key, value)
sc = SparkContext.getOrCreate(sparkConf)
session = SparkSession(sc)
for key, value in self._options.items(): for key, value in self._options.items():
sparkConf.set(key, value) session.conf.set(key, value)
sparkContext = SparkContext.getOrCreate(sparkConf) return session
return SQLContext.getOrCreate(sparkContext).sparkSession
builder = Builder() builder = Builder()
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment