-
- Downloads
[SPARK-19849][SQL] Support ArrayType in to_json to produce JSON array
## What changes were proposed in this pull request? This PR proposes to support an array of struct type in `to_json` as below: ```scala import org.apache.spark.sql.functions._ val df = Seq(Tuple1(Tuple1(1) :: Nil)).toDF("a") df.select(to_json($"a").as("json")).show() ``` ``` +----------+ | json| +----------+ |[{"_1":1}]| +----------+ ``` Currently, it throws an exception as below (a newline manually inserted for readability): ``` org.apache.spark.sql.AnalysisException: cannot resolve 'structtojson(`array`)' due to data type mismatch: structtojson requires that the expression is a struct expression.;; ``` This allows the roundtrip with `from_json` as below: ```scala import org.apache.spark.sql.functions._ import org.apache.spark.sql.types._ val schema = ArrayType(StructType(StructField("a", IntegerType) :: Nil)) val df = Seq("""[{"a":1}, {"a":2}]""").toDF("json").select(from_json($"json", schema).as("array")) df.show() // Read back. df.select(to_json($"array").as("json")).show() ``` ``` +----------+ | array| +----------+ |[[1], [2]]| +----------+ +-----------------+ | json| +-----------------+ |[{"a":1},{"a":2}]| +-----------------+ ``` Also, this PR proposes to rename from `StructToJson` to `StructsToJson ` and `JsonToStruct` to `JsonToStructs`. ## How was this patch tested? Unit tests in `JsonFunctionsSuite` and `JsonExpressionsSuite` for Scala, doctest for Python and test in `test_sparkSQL.R` for R. Author: hyukjinkwon <gurwls223@gmail.com> Closes #17192 from HyukjinKwon/SPARK-19849.
Showing
- R/pkg/R/functions.R 12 additions, 6 deletionsR/pkg/R/functions.R
- R/pkg/inst/tests/testthat/test_sparkSQL.R 4 additions, 0 deletionsR/pkg/inst/tests/testthat/test_sparkSQL.R
- python/pyspark/sql/functions.py 10 additions, 5 deletionspython/pyspark/sql/functions.py
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala 2 additions, 2 deletions...apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala 45 additions, 25 deletions...ache/spark/sql/catalyst/expressions/jsonExpressions.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala 14 additions, 9 deletions...org/apache/spark/sql/catalyst/json/JacksonGenerator.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/JsonExpressionsSuite.scala 52 additions, 25 deletions...spark/sql/catalyst/expressions/JsonExpressionsSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/functions.scala 19 additions, 15 deletionssql/core/src/main/scala/org/apache/spark/sql/functions.scala
- sql/core/src/test/resources/sql-tests/inputs/json-functions.sql 1 addition, 0 deletions...re/src/test/resources/sql-tests/inputs/json-functions.sql
- sql/core/src/test/resources/sql-tests/results/json-functions.sql.out 53 additions, 43 deletions...c/test/resources/sql-tests/results/json-functions.sql.out
- sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala 24 additions, 2 deletions.../test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala
Loading
Please register or sign in to comment