-
- Downloads
[SPARK-16609] Add to_date/to_timestamp with format functions
## What changes were proposed in this pull request? This pull request adds two new user facing functions: - `to_date` which accepts an expression and a format and returns a date. - `to_timestamp` which accepts an expression and a format and returns a timestamp. For example, Given a date in format: `2016-21-05`. (YYYY-dd-MM) ### Date Function *Previously* ``` to_date(unix_timestamp(lit("2016-21-05"), "yyyy-dd-MM").cast("timestamp")) ``` *Current* ``` to_date(lit("2016-21-05"), "yyyy-dd-MM") ``` ### Timestamp Function *Previously* ``` unix_timestamp(lit("2016-21-05"), "yyyy-dd-MM").cast("timestamp") ``` *Current* ``` to_timestamp(lit("2016-21-05"), "yyyy-dd-MM") ``` ### Tasks - [X] Add `to_date` to Scala Functions - [x] Add `to_date` to Python Functions - [x] Add `to_date` to SQL Functions - [X] Add `to_timestamp` to Scala Functions - [x] Add `to_timestamp` to Python Functions - [x] Add `to_timestamp` to SQL Functions - [x] Add function to R ## How was this patch tested? - [x] Add Functions to `DateFunctionsSuite` - Test new `ParseToTimestamp` Expression (*not necessary*) - Test new `ParseToDate` Expression (*not necessary*) - [x] Add test for R - [x] Add test for Python in test.py Please review http://spark.apache.org/contributing.html before opening a pull request. Author: anabranch <wac.chambers@gmail.com> Author: Bill Chambers <bill@databricks.com> Author: anabranch <bill@databricks.com> Closes #16138 from anabranch/SPARK-16609.
Showing
- R/pkg/NAMESPACE 1 addition, 0 deletionsR/pkg/NAMESPACE
- R/pkg/R/functions.R 73 additions, 7 deletionsR/pkg/R/functions.R
- R/pkg/R/generics.R 5 additions, 1 deletionR/pkg/R/generics.R
- R/pkg/inst/tests/testthat/test_sparkSQL.R 1 addition, 0 deletionsR/pkg/inst/tests/testthat/test_sparkSQL.R
- python/pyspark/sql/functions.py 46 additions, 6 deletionspython/pyspark/sql/functions.py
- python/pyspark/sql/tests.py 9 additions, 0 deletionspython/pyspark/sql/tests.py
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala 2 additions, 1 deletion...apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala 64 additions, 0 deletions.../spark/sql/catalyst/expressions/datetimeExpressions.scala
- sql/core/src/main/scala/org/apache/spark/sql/functions.scala 33 additions, 0 deletionssql/core/src/main/scala/org/apache/spark/sql/functions.scala
- sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala 61 additions, 7 deletions.../test/scala/org/apache/spark/sql/DateFunctionsSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/catalyst/ExpressionToSQLSuite.scala 1 addition, 0 deletions.../org/apache/spark/sql/catalyst/ExpressionToSQLSuite.scala
Loading
Please register or sign in to comment