-
- Downloads
[SPARK-15709][SQL] Prevent `freqItems` from raising `UnsupportedOperationException: empty.min`
## What changes were proposed in this pull request? Currently, `freqItems` raises `UnsupportedOperationException` on `empty.min` usually when its `support` argument is high. ```scala scala> spark.createDataset(Seq(1, 2, 2, 3, 3, 3)).stat.freqItems(Seq("value"), 2) 16/06/01 11:11:38 ERROR Executor: Exception in task 5.0 in stage 0.0 (TID 5) java.lang.UnsupportedOperationException: empty.min ... ``` Also, the parameter checking message is wrong. ``` require(support >= 1e-4, s"support ($support) must be greater than 1e-4.") ``` This PR changes the logic to handle the `empty` case and also improves parameter checking. ## How was this patch tested? Pass the Jenkins tests (with a new testcase). Author: Dongjoon Hyun <dongjoon@apache.org> Closes #13449 from dongjoon-hyun/SPARK-15709.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/stat/FrequentItems.scala 2 additions, 2 deletions...a/org/apache/spark/sql/execution/stat/FrequentItems.scala
- sql/core/src/test/scala/org/apache/spark/sql/DataFrameStatSuite.scala 11 additions, 0 deletions.../test/scala/org/apache/spark/sql/DataFrameStatSuite.scala
Loading
Please register or sign in to comment