-
- Downloads
[SPARK-18191][CORE][FOLLOWUP] Call `setConf` if `OutputFormat` is `Configurable`.
## What changes were proposed in this pull request? We should call `setConf` if `OutputFormat` is `Configurable`, this should be done before we create `OutputCommitter` and `RecordWriter`. This is follow up of #15769, see discussion [here](https://github.com/apache/spark/pull/15769/files#r87064229) ## How was this patch tested? Add test of this case in `PairRDDFunctionsSuite`. Author: jiangxingbo <jiangxb1987@gmail.com> Closes #15823 from jiangxb1987/config-format.
Showing
- core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala 8 additions, 1 deletion...che/spark/internal/io/HadoopMapReduceCommitProtocol.scala
- core/src/main/scala/org/apache/spark/internal/io/SparkHadoopMapReduceWriter.scala 7 additions, 2 deletions...apache/spark/internal/io/SparkHadoopMapReduceWriter.scala
- core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala 15 additions, 0 deletions...st/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala
Loading
Please register or sign in to comment