-
- Downloads
[SPARK-13668][SQL] Reorder filter/join predicates to short-circuit isNotNull checks
## What changes were proposed in this pull request? If a filter predicate or a join condition consists of `IsNotNull` checks, we should reorder these checks such that these non-nullability checks are evaluated before the rest of the predicates. For e.g., if a filter predicate is of the form `a > 5 && isNotNull(b)`, we should rewrite this as `isNotNull(b) && a > 5` during physical plan generation. ## How was this patch tested? new unit tests that verify the physical plan for both filters and joins in `ReorderedPredicateSuite` Author: Sameer Agarwal <sameer@databricks.com> Closes #11511 from sameeragarwal/reorder-isnotnull.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/QueryPlanner.scala 23 additions, 1 deletion...org/apache/spark/sql/catalyst/planning/QueryPlanner.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala 24 additions, 13 deletions...cala/org/apache/spark/sql/execution/SparkStrategies.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/ReorderedPredicateSuite.scala 103 additions, 0 deletions.../apache/spark/sql/execution/ReorderedPredicateSuite.scala
Loading
Please register or sign in to comment