-
- Downloads
[SPARK-13989] [SQL] Remove non-vectorized/unsafe-row parquet record reader
## What changes were proposed in this pull request? This PR cleans up the new parquet record reader with the following changes: 1. Removes the non-vectorized parquet reader code from `UnsafeRowParquetRecordReader`. 2. Removes the non-vectorized column reader code from `ColumnReader`. 3. Renames `UnsafeRowParquetRecordReader` to `VectorizedParquetRecordReader` and `ColumnReader` to `VectorizedColumnReader` 4. Deprecate `PARQUET_UNSAFE_ROW_RECORD_READER_ENABLED` ## How was this patch tested? Refactoring only; Existing tests should reveal any problems. Author: Sameer Agarwal <sameer@databricks.com> Closes #11799 from sameeragarwal/vectorized-parquet.
Showing
- sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java 37 additions, 282 deletions...on/datasources/parquet/VectorizedParquetRecordReader.java
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 9 additions, 0 deletions...ala/org/apache/spark/sql/execution/command/commands.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SqlNewHadoopRDD.scala 8 additions, 12 deletions...che/spark/sql/execution/datasources/SqlNewHadoopRDD.scala
- sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 1 addition, 5 deletions...rc/main/scala/org/apache/spark/sql/internal/SQLConf.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetEncodingSuite.scala 2 additions, 2 deletions.../execution/datasources/parquet/ParquetEncodingSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala 3 additions, 3 deletions...ql/execution/datasources/parquet/ParquetFilterSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala 7 additions, 7 deletions...rk/sql/execution/datasources/parquet/ParquetIOSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadBenchmark.scala 8 additions, 53 deletions.../execution/datasources/parquet/ParquetReadBenchmark.scala
Loading
Please register or sign in to comment