-
- Downloads
[SPARK-2406][SQL] Initial support for using ParquetTableScan to read HiveMetaStore tables.
This PR adds an experimental flag `spark.sql.hive.convertMetastoreParquet` that when true causes the planner to detects tables that use Hive's Parquet SerDe and instead plans them using Spark SQL's native `ParquetTableScan`. Author: Michael Armbrust <michael@databricks.com> Author: Yin Huai <huai@cse.ohio-state.edu> Closes #1819 from marmbrus/parquetMetastore and squashes the following commits: 1620079 [Michael Armbrust] Revert "remove hive parquet bundle" cc30430 [Michael Armbrust] Merge remote-tracking branch 'origin/master' into parquetMetastore 4f3d54f [Michael Armbrust] fix style 41ebc5f [Michael Armbrust] remove hive parquet bundle a43e0da [Michael Armbrust] Merge remote-tracking branch 'origin/master' into parquetMetastore 4c4dc19 [Michael Armbrust] Fix bug with tree splicing. ebb267e [Michael Armbrust] include parquet hive to tests pass (Remove this later). c0d9b72 [Michael Armbrust] Avoid creating a HadoopRDD per partition. Add dirty hacks to retrieve partition values from the InputSplit. 8cdc93c [Michael Armbrust] Merge pull request #8 from yhuai/parquetMetastore a0baec7 [Yin Huai] Partitioning columns can be resolved. 1161338 [Michael Armbrust] Add a test to make sure conversion is actually happening 212d5cd [Michael Armbrust] Initial support for using ParquetTableScan to read HiveMetaStore tables.
Showing
- project/SparkBuild.scala 0 additions, 1 deletionproject/SparkBuild.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala 12 additions, 0 deletions...scala/org/apache/spark/sql/execution/basicOperators.scala
- sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetRelation.scala 5 additions, 3 deletions.../scala/org/apache/spark/sql/parquet/ParquetRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTableOperations.scala 57 additions, 17 deletions...org/apache/spark/sql/parquet/ParquetTableOperations.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala 9 additions, 0 deletions...rc/main/scala/org/apache/spark/sql/hive/HiveContext.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala 117 additions, 2 deletions...main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/parquet/FakeParquetSerDe.scala 56 additions, 0 deletions.../org/apache/spark/sql/hive/parquet/FakeParquetSerDe.scala
- sql/hive/src/test/scala/org/apache/spark/sql/parquet/ParquetMetastoreSuite.scala 171 additions, 0 deletions.../org/apache/spark/sql/parquet/ParquetMetastoreSuite.scala
Loading
Please register or sign in to comment