Skip to content
Snippets Groups Projects
Commit 2d2411fa authored by Xiangrui Meng's avatar Xiangrui Meng
Browse files

[SPARK-11672][ML] Set active SQLContext in MLlibTestSparkContext.beforeAll

Still saw some error messages caused by `SQLContext.getOrCreate`:

https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-SBT/3997/AMPLAB_JENKINS_BUILD_PROFILE=hadoop2.3,label=spark-test/testReport/junit/org.apache.spark.ml.util/JavaDefaultReadWriteSuite/testDefaultReadWrite/

This PR sets the active SQLContext in beforeAll, which is not automatically set in `new SQLContext`. This makes `SQLContext.getOrCreate` return the right SQLContext.

cc: yhuai

Author: Xiangrui Meng <meng@databricks.com>

Closes #9694 from mengxr/SPARK-11672.3.
parent d7b2b97a
No related branches found
No related tags found
No related merge requests found
...@@ -48,8 +48,11 @@ private[util] sealed trait BaseReadWrite { ...@@ -48,8 +48,11 @@ private[util] sealed trait BaseReadWrite {
/** /**
* Returns the user-specified SQL context or the default. * Returns the user-specified SQL context or the default.
*/ */
protected final def sqlContext: SQLContext = optionSQLContext.getOrElse { protected final def sqlContext: SQLContext = {
SQLContext.getOrCreate(SparkContext.getOrCreate()) if (optionSQLContext.isEmpty) {
optionSQLContext = Some(SQLContext.getOrCreate(SparkContext.getOrCreate()))
}
optionSQLContext.get
} }
/** Returns the [[SparkContext]] underlying [[sqlContext]] */ /** Returns the [[SparkContext]] underlying [[sqlContext]] */
......
...@@ -34,6 +34,7 @@ trait MLlibTestSparkContext extends BeforeAndAfterAll { self: Suite => ...@@ -34,6 +34,7 @@ trait MLlibTestSparkContext extends BeforeAndAfterAll { self: Suite =>
sc = new SparkContext(conf) sc = new SparkContext(conf)
SQLContext.clearActive() SQLContext.clearActive()
sqlContext = new SQLContext(sc) sqlContext = new SQLContext(sc)
SQLContext.setActive(sqlContext)
} }
override def afterAll() { override def afterAll() {
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment