-
- Downloads
[SPARK-20586][SQL] Add deterministic to ScalaUDF
### What changes were proposed in this pull request? Like [Hive UDFType](https://hive.apache.org/javadocs/r2.0.1/api/org/apache/hadoop/hive/ql/udf/UDFType.html), we should allow users to add the extra flags for ScalaUDF and JavaUDF too. _stateful_/_impliesOrder_ are not applicable to our Scala UDF. Thus, we only add the following two flags. - deterministic: Certain optimizations should not be applied if UDF is not deterministic. Deterministic UDF returns same result each time it is invoked with a particular input. This determinism just needs to hold within the context of a query. When the deterministic flag is not correctly set, the results could be wrong. For ScalaUDF in Dataset APIs, users can call the following extra APIs for `UserDefinedFunction` to make the corresponding changes. - `nonDeterministic`: Updates UserDefinedFunction to non-deterministic. Also fixed the Java UDF name loss issue. Will submit a separate PR for `distinctLike` for UDAF ### How was this patch tested? Added test cases for both ScalaUDF Author: gatorsmile <gatorsmile@gmail.com> Author: Wenchen Fan <cloud0fan@gmail.com> Closes #17848 from gatorsmile/udfRegister.
Showing
- python/pyspark/sql/context.py 2 additions, 2 deletionspython/pyspark/sql/context.py
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala 1 addition, 1 deletion...ala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala 7 additions, 3 deletions.../org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
- sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala 139 additions, 104 deletions...src/main/scala/org/apache/spark/sql/UDFRegistration.scala
- sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala 35 additions, 13 deletions...rg/apache/spark/sql/expressions/UserDefinedFunction.scala
- sql/core/src/main/scala/org/apache/spark/sql/functions.scala 75 additions, 38 deletionssql/core/src/main/scala/org/apache/spark/sql/functions.scala
- sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala 19 additions, 3 deletionssql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala
Loading
Please register or sign in to comment