-
- Downloads
[SPARK-12616][SQL] Making Logical Operator `Union` Support Arbitrary Number of Children
The existing `Union` logical operator only supports two children. Thus, adding a new logical operator `Unions` which can have arbitrary number of children to replace the existing one. `Union` logical plan is a binary node. However, a typical use case for union is to union a very large number of input sources (DataFrames, RDDs, or files). It is not uncommon to union hundreds of thousands of files. In this case, our optimizer can become very slow due to the large number of logical unions. We should change the Union logical plan to support an arbitrary number of children, and add a single rule in the optimizer to collapse all adjacent `Unions` into a single `Unions`. Note that this problem doesn't exist in physical plan, because the physical `Unions` already supports arbitrary number of children. Author: gatorsmile <gatorsmile@gmail.com> Author: xiaoli <lixiao1983@gmail.com> Author: Xiao Li <xiaoli@Xiaos-MacBook-Pro.local> Closes #10577 from gatorsmile/unionAllMultiChildren.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystQl.scala 2 additions, 2 deletions...main/scala/org/apache/spark/sql/catalyst/CatalystQl.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala 11 additions, 1 deletion...ala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala 8 additions, 0 deletions...rg/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala 51 additions, 20 deletions...apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala 46 additions, 25 deletions...a/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala 19 additions, 4 deletions...ala/org/apache/spark/sql/catalyst/planning/patterns.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala 34 additions, 13 deletions...che/spark/sql/catalyst/plans/logical/basicOperators.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala 6 additions, 0 deletions...rg/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecisionSuite.scala 1 addition, 1 deletion...e/spark/sql/catalyst/analysis/DecimalPrecisionSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercionSuite.scala 66 additions, 16 deletions...e/spark/sql/catalyst/analysis/HiveTypeCoercionSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/SetOperationSuite.scala 34 additions, 9 deletions...ache/spark/sql/catalyst/optimizer/SetOperationSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala 4 additions, 1 deletionsql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala
- sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala 7 additions, 2 deletionssql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala 1 addition, 1 deletion...cala/org/apache/spark/sql/execution/SparkStrategies.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala 4 additions, 7 deletions...scala/org/apache/spark/sql/execution/basicOperators.scala
- sql/core/src/test/java/test/org/apache/spark/sql/JavaDatasetSuite.java 2 additions, 2 deletions...test/java/test/org/apache/spark/sql/JavaDatasetSuite.java
- sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala 15 additions, 1 deletion.../src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala 0 additions, 12 deletions...t/scala/org/apache/spark/sql/execution/PlannerSuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/SQLBuilder.scala 7 additions, 5 deletions...src/main/scala/org/apache/spark/sql/hive/SQLBuilder.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/LogicalPlanToSQLSuite.scala 4 additions, 0 deletions...ala/org/apache/spark/sql/hive/LogicalPlanToSQLSuite.scala
Loading
Please register or sign in to comment