-
- Downloads
[SPARK-21092][SQL] Wire SQLConf in logical plan and expressions
## What changes were proposed in this pull request? It is really painful to not have configs in logical plan and expressions. We had to add all sorts of hacks (e.g. pass SQLConf explicitly in functions). This patch exposes SQLConf in logical plan, using a thread local variable and a getter closure that's set once there is an active SparkSession. The implementation is a bit of a hack, since we didn't anticipate this need in the beginning (config was only exposed in physical plan). The implementation is described in `SQLConf.get`. In terms of future work, we should follow up to clean up CBO (remove the need for passing in config). ## How was this patch tested? Updated relevant tests for constraint propagation. Author: Reynold Xin <rxin@databricks.com> Closes #18299 from rxin/SPARK-21092.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala 13 additions, 12 deletions...a/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala 2 additions, 3 deletions...scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala 3 additions, 0 deletions...scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlanConstraints.scala 11 additions, 22 deletions...pache/spark/sql/catalyst/plans/QueryPlanConstraints.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 42 additions, 0 deletions...rc/main/scala/org/apache/spark/sql/internal/SQLConf.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/BinaryComparisonSimplificationSuite.scala 1 addition, 1 deletion...alyst/optimizer/BinaryComparisonSimplificationSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/BooleanSimplificationSuite.scala 1 addition, 1 deletion...k/sql/catalyst/optimizer/BooleanSimplificationSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/InferFiltersFromConstraintsSuite.scala 10 additions, 14 deletions...catalyst/optimizer/InferFiltersFromConstraintsSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OuterJoinEliminationSuite.scala 17 additions, 20 deletions...rk/sql/catalyst/optimizer/OuterJoinEliminationSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/PropagateEmptyRelationSuite.scala 2 additions, 2 deletions.../sql/catalyst/optimizer/PropagateEmptyRelationSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/PruneFiltersSuite.scala 15 additions, 21 deletions...ache/spark/sql/catalyst/optimizer/PruneFiltersSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/SetOperationSuite.scala 1 addition, 1 deletion...ache/spark/sql/catalyst/optimizer/SetOperationSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/ConstraintPropagationSuite.scala 18 additions, 11 deletions...spark/sql/catalyst/plans/ConstraintPropagationSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 5 additions, 0 deletions...re/src/main/scala/org/apache/spark/sql/SparkSession.scala
Loading
Please register or sign in to comment