-
- Downloads
[SPARK-7562][SPARK-6444][SQL] Improve error reporting for expression data type mismatch
It seems hard to find a common pattern of checking types in `Expression`. Sometimes we know what input types we need(like `And`, we know we need two booleans), sometimes we just have some rules(like `Add`, we need 2 numeric types which are equal). So I defined a general interface `checkInputDataTypes` in `Expression` which returns a `TypeCheckResult`. `TypeCheckResult` can tell whether this expression passes the type checking or what the type mismatch is. This PR mainly works on apply input types checking for arithmetic and predicate expressions. TODO: apply type checking interface to more expressions. Author: Wenchen Fan <cloud0fan@outlook.com> Closes #6405 from cloud-fan/6444 and squashes the following commits: b5ff31b [Wenchen Fan] address comments b917275 [Wenchen Fan] rebase 39929d9 [Wenchen Fan] add todo 0808fd2 [Wenchen Fan] make constrcutor of TypeCheckResult private 3bee157 [Wenchen Fan] and decimal type coercion rule for binary comparison 8883025 [Wenchen Fan] apply type check interface to CaseWhen cffb67c [Wenchen Fan] to have resolved call the data type check function 6eaadff [Wenchen Fan] add equal type constraint to EqualTo 3affbd8 [Wenchen Fan] more fixes 654d46a [Wenchen Fan] improve tests e0a3628 [Wenchen Fan] improve error message 1524ff6 [Wenchen Fan] fix style 69ca3fe [Wenchen Fan] add error message and tests c71d02c [Wenchen Fan] fix hive tests 6491721 [Wenchen Fan] use value class TypeCheckResult 7ae76b9 [Wenchen Fan] address comments cb77e4f [Wenchen Fan] Improve error reporting for expression data type mismatch
Showing
- core/src/test/scala/org/apache/spark/SparkFunSuite.scala 2 additions, 2 deletionscore/src/test/scala/org/apache/spark/SparkFunSuite.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala 7 additions, 5 deletions...rg/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala 61 additions, 71 deletions...apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCheckResult.scala 45 additions, 0 deletions.../apache/spark/sql/catalyst/analysis/TypeCheckResult.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala 22 additions, 6 deletions...rg/apache/spark/sql/catalyst/expressions/Expression.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala 122 additions, 186 deletions...rg/apache/spark/sql/catalyst/expressions/arithmetic.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathfuncs/binary.scala 4 additions, 13 deletions...che/spark/sql/catalyst/expressions/mathfuncs/binary.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala 103 additions, 123 deletions...rg/apache/spark/sql/catalyst/expressions/predicates.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala 4 additions, 0 deletions...a/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateUtils.scala 1 addition, 1 deletion.../scala/org/apache/spark/sql/catalyst/util/DateUtils.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TypeUtils.scala 56 additions, 0 deletions.../scala/org/apache/spark/sql/catalyst/util/TypeUtils.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataType.scala 1 addition, 1 deletion.../src/main/scala/org/apache/spark/sql/types/DataType.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecisionSuite.scala 4 additions, 2 deletions...e/spark/sql/catalyst/analysis/DecimalPrecisionSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercionSuite.scala 6 additions, 9 deletions...e/spark/sql/catalyst/analysis/HiveTypeCoercionSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionTypeCheckingSuite.scala 143 additions, 0 deletions...ql/catalyst/expressions/ExpressionTypeCheckingSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/json/InferSchema.scala 1 addition, 1 deletion...rc/main/scala/org/apache/spark/sql/json/InferSchema.scala
- sql/core/src/main/scala/org/apache/spark/sql/json/JsonRDD.scala 1 addition, 1 deletion...re/src/main/scala/org/apache/spark/sql/json/JsonRDD.scala
Loading
Please register or sign in to comment