-
- Downloads
[SPARK-11727][SQL] Split ExpressionEncoder into FlatEncoder and ProductEncoder
also add more tests for encoders, and fix bugs that I found: * when convert array to catalyst array, we can only skip element conversion for native types(e.g. int, long, boolean), not `AtomicType`(String is AtomicType but we need to convert it) * we should also handle scala `BigDecimal` when convert from catalyst `Decimal`. * complex map type should be supported other issues that still in investigation: * encode java `BigDecimal` and decode it back, seems we will loss precision info. * when encode case class that defined inside a object, `ClassNotFound` exception will be thrown. I'll remove unused code in a follow-up PR. Author: Wenchen Fan <wenchen@databricks.com> Closes #9693 from cloud-fan/split.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala 1 addition, 1 deletion...scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/FlatEncoder.scala 50 additions, 0 deletions.../org/apache/spark/sql/catalyst/encoders/FlatEncoder.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ProductEncoder.scala 452 additions, 0 deletions...g/apache/spark/sql/catalyst/encoders/ProductEncoder.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala 16 additions, 42 deletions...a/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala 1 addition, 1 deletion...la/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/GenericArrayData.scala 1 addition, 1 deletion...org/apache/spark/sql/catalyst/util/GenericArrayData.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala 31 additions, 228 deletions.../spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/FlatEncoderSuite.scala 74 additions, 0 deletions...apache/spark/sql/catalyst/encoders/FlatEncoderSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ProductEncoderSuite.scala 123 additions, 0 deletions...che/spark/sql/catalyst/encoders/ProductEncoderSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/GroupedDataset.scala 2 additions, 5 deletions.../src/main/scala/org/apache/spark/sql/GroupedDataset.scala
- sql/core/src/main/scala/org/apache/spark/sql/SQLImplicits.scala 13 additions, 9 deletions...re/src/main/scala/org/apache/spark/sql/SQLImplicits.scala
- sql/core/src/main/scala/org/apache/spark/sql/functions.scala 2 additions, 2 deletionssql/core/src/main/scala/org/apache/spark/sql/functions.scala
Loading
Please register or sign in to comment