-
- Downloads
[Spark-1461] Deferred Expression Evaluation (short-circuit evaluation)
This patch unify the foldable & nullable interface for Expression. 1) Deterministic-less UDF (like Rand()) can not be folded. 2) Short-circut will significantly improves the performance in Expression Evaluation, however, the stateful UDF should not be ignored in a short-circuit evaluation(e.g. in expression: col1 > 0 and row_sequence() < 1000, row_sequence() can not be ignored even if col1 > 0 is false) I brought an concept of DeferredObject from Hive, which has 2 kinds of children classes (EagerResult / DeferredResult), the former requires triggering the evaluation before it's created, while the later trigger the evaluation when first called its get() method. Author: Cheng Hao <hao.cheng@intel.com> Closes #446 from chenghao-intel/expression_deferred_evaluation and squashes the following commits: d2729de [Cheng Hao] Fix the codestyle issues a08f09c [Cheng Hao] fix bug in or/and short-circuit evaluation af2236b [Cheng Hao] revert the short-circuit expression evaluation for IF b7861d2 [Cheng Hao] Add Support for Deferred Expression Evaluation
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala 32 additions, 15 deletions...rg/apache/spark/sql/catalyst/expressions/predicates.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUdfs.scala 21 additions, 7 deletions...e/src/main/scala/org/apache/spark/sql/hive/hiveUdfs.scala
Loading
Please register or sign in to comment