-
- Downloads
[SPARK-15494][SQL] encoder code cleanup
## What changes were proposed in this pull request? Our encoder framework has been evolved a lot, this PR tries to clean up the code to make it more readable and emphasise the concept that encoder should be used as a container of serde expressions. 1. move validation logic to analyzer instead of encoder 2. only have a `resolveAndBind` method in encoder instead of `resolve` and `bind`, as we don't have the encoder life cycle concept anymore. 3. `Dataset` don't need to keep a resolved encoder, as there is no such concept anymore. bound encoder is still needed to do serialization outside of query framework. 4. Using `BoundReference` to represent an unresolved field in deserializer expression is kind of weird, this PR adds a `GetColumnByOrdinal` for this purpose. (serializer expression still use `BoundReference`, we can replace it with `GetColumnByOrdinal` in follow-ups) ## How was this patch tested? existing test Author: Wenchen Fan <wenchen@databricks.com> Author: Cheng Lian <lian@databricks.com> Closes #13269 from cloud-fan/clean-encoder.
Showing
- mllib/src/test/scala/org/apache/spark/mllib/linalg/UDTSerializationBenchmark.scala 1 addition, 1 deletion...apache/spark/mllib/linalg/UDTSerializationBenchmark.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/Encoders.scala 2 additions, 1 deletion...talyst/src/main/scala/org/apache/spark/sql/Encoders.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/JavaTypeInference.scala 3 additions, 3 deletions...ala/org/apache/spark/sql/catalyst/JavaTypeInference.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala 153 additions, 154 deletions...scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala 49 additions, 4 deletions...ala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala 7 additions, 0 deletions...a/org/apache/spark/sql/catalyst/analysis/unresolved.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala 27 additions, 107 deletions...pache/spark/sql/catalyst/encoders/ExpressionEncoder.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala 2 additions, 6 deletions...a/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala 3 additions, 16 deletions.../org/apache/spark/sql/catalyst/plans/logical/object.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/EncoderResolutionSuite.scala 17 additions, 25 deletions.../spark/sql/catalyst/encoders/EncoderResolutionSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala 4 additions, 7 deletions.../spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/RowEncoderSuite.scala 7 additions, 7 deletions.../apache/spark/sql/catalyst/encoders/RowEncoderSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala 24 additions, 24 deletionssql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
- sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala 9 additions, 17 deletions...n/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala
- sql/core/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala 1 addition, 1 deletion...scala/org/apache/spark/sql/RelationalGroupedDataset.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TypedAggregateExpression.scala 3 additions, 3 deletions...rk/sql/execution/aggregate/TypedAggregateExpression.scala
- sql/core/src/main/scala/org/apache/spark/sql/functions.scala 1 addition, 1 deletionsql/core/src/main/scala/org/apache/spark/sql/functions.scala
- sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala 2 additions, 6 deletions...re/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala 4 additions, 4 deletionssql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/GroupedIteratorSuite.scala 3 additions, 3 deletions...org/apache/spark/sql/execution/GroupedIteratorSuite.scala
Loading
Please register or sign in to comment