-
- Downloads
[SPARK-18774][CORE][SQL] Ignore non-existing files when ignoreCorruptFiles is enabled
## What changes were proposed in this pull request? When `ignoreCorruptFiles` is enabled, it's better to also ignore non-existing files. ## How was this patch tested? Jenkins Author: Shixiong Zhu <shixiong@databricks.com> Closes #16203 from zsxwing/ignore-file-not-found.
Showing
- core/src/main/scala/org/apache/spark/internal/config/package.scala 2 additions, 1 deletion...main/scala/org/apache/spark/internal/config/package.scala
- core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala 11 additions, 3 deletionscore/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala
- core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala 15 additions, 4 deletionscore/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala 3 additions, 0 deletions.../apache/spark/sql/execution/datasources/FileScanRDD.scala
- sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 2 additions, 1 deletion...rc/main/scala/org/apache/spark/sql/internal/SQLConf.scala
Loading
Please register or sign in to comment