-
- Downloads
[SPARK-21440][SQL][PYSPARK] Refactor ArrowConverters and add ArrayType and StructType support.
## What changes were proposed in this pull request? This is a refactoring of `ArrowConverters` and related classes. 1. Refactor `ColumnWriter` as `ArrowWriter`. 2. Add `ArrayType` and `StructType` support. 3. Refactor `ArrowConverters` to skip intermediate `ArrowRecordBatch` creation. ## How was this patch tested? Added some tests and existing tests. Author: Takuya UESHIN <ueshin@databricks.com> Closes #18655 from ueshin/issues/SPARK-21440.
Showing
- python/pyspark/sql/tests.py 2 additions, 2 deletionspython/pyspark/sql/tests.py
- sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala 3 additions, 1 deletionsql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala 41 additions, 310 deletions...rg/apache/spark/sql/execution/arrow/ArrowConverters.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriter.scala 323 additions, 0 deletions...la/org/apache/spark/sql/execution/arrow/ArrowWriter.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowConvertersSuite.scala 445 additions, 2 deletions...ache/spark/sql/execution/arrow/ArrowConvertersSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowWriterSuite.scala 260 additions, 0 deletions...g/apache/spark/sql/execution/arrow/ArrowWriterSuite.scala
Loading
Please register or sign in to comment