-
- Downloads
[SPARK-16289][SQL] Implement posexplode table generating function
## What changes were proposed in this pull request? This PR implements `posexplode` table generating function. Currently, master branch raises the following exception for `map` argument. It's different from Hive. **Before** ```scala scala> sql("select posexplode(map('a', 1, 'b', 2))").show org.apache.spark.sql.AnalysisException: No handler for Hive UDF ... posexplode() takes an array as a parameter; line 1 pos 7 ``` **After** ```scala scala> sql("select posexplode(map('a', 1, 'b', 2))").show +---+---+-----+ |pos|key|value| +---+---+-----+ | 0| a| 1| | 1| b| 2| +---+---+-----+ ``` For `array` argument, `after` is the same with `before`. ``` scala> sql("select posexplode(array(1, 2, 3))").show +---+---+ |pos|col| +---+---+ | 0| 1| | 1| 2| | 2| 3| +---+---+ ``` ## How was this patch tested? Pass the Jenkins tests with newly added testcases. Author: Dongjoon Hyun <dongjoon@apache.org> Closes #13971 from dongjoon-hyun/SPARK-16289.
Showing
- R/pkg/NAMESPACE 1 addition, 0 deletionsR/pkg/NAMESPACE
- R/pkg/R/functions.R 17 additions, 0 deletionsR/pkg/R/functions.R
- R/pkg/R/generics.R 4 additions, 0 deletionsR/pkg/R/generics.R
- R/pkg/inst/tests/testthat/test_sparkSQL.R 1 addition, 1 deletionR/pkg/inst/tests/testthat/test_sparkSQL.R
- python/pyspark/sql/functions.py 21 additions, 0 deletionspython/pyspark/sql/functions.py
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala 1 addition, 0 deletions...apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala 56 additions, 10 deletions...rg/apache/spark/sql/catalyst/expressions/generators.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ExpressionTypeCheckingSuite.scala 2 additions, 0 deletions...k/sql/catalyst/analysis/ExpressionTypeCheckingSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/GeneratorExpressionSuite.scala 71 additions, 0 deletions...k/sql/catalyst/expressions/GeneratorExpressionSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/Column.scala 1 addition, 0 deletionssql/core/src/main/scala/org/apache/spark/sql/Column.scala
- sql/core/src/main/scala/org/apache/spark/sql/functions.scala 8 additions, 0 deletionssql/core/src/main/scala/org/apache/spark/sql/functions.scala
- sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala 0 additions, 60 deletions...st/scala/org/apache/spark/sql/ColumnExpressionSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/GeneratorFunctionSuite.scala 92 additions, 0 deletions...t/scala/org/apache/spark/sql/GeneratorFunctionSuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala 1 addition, 1 deletion.../scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala
Loading
Please register or sign in to comment