-
- Downloads
[SPARK-15140][SQL] make the semantics of null input object for encoder clear
## What changes were proposed in this pull request? For input object of non-flat type, we can't encode it to row if it's null, as Spark SQL doesn't allow row to be null, only its columns can be null. This PR explicitly add this constraint and throw exception if users break it. ## How was this patch tested? several new tests Author: Wenchen Fan <wenchen@databricks.com> Closes #13469 from cloud-fan/null-object.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala 10 additions, 3 deletions...pache/spark/sql/catalyst/encoders/ExpressionEncoder.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala 3 additions, 4 deletions...a/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala 2 additions, 2 deletions...ache/spark/sql/catalyst/expressions/objects/objects.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/RowEncoderSuite.scala 8 additions, 0 deletions.../apache/spark/sql/catalyst/encoders/RowEncoderSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala 10 additions, 0 deletions...re/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
Loading
Please register or sign in to comment