[SPARK-23389][CORE] When the shuffle dependency specifies aggregation ,and...
[SPARK-23389][CORE] When the shuffle dependency specifies aggregation ,and `dependency.mapSideCombine =false`, we should be able to use serialized sorting. ## What changes were proposed in this pull request? When the shuffle dependency specifies aggregation ,and `dependency.mapSideCombine=false`, in the map side,there is no need for aggregation and sorting, so we should be able to use serialized sorting. ## How was this patch tested? Existing unit test Author: liuxian <liu.xian3@zte.com.cn> Closes #20576 from 10110346/mapsidecombine.
Showing
- core/src/main/scala/org/apache/spark/Dependency.scala 3 additions, 0 deletionscore/src/main/scala/org/apache/spark/Dependency.scala
- core/src/main/scala/org/apache/spark/shuffle/BlockStoreShuffleReader.scala 0 additions, 1 deletion...la/org/apache/spark/shuffle/BlockStoreShuffleReader.scala
- core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleManager.scala 3 additions, 3 deletions...la/org/apache/spark/shuffle/sort/SortShuffleManager.scala
- core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala 0 additions, 2 deletions...ala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala
- core/src/test/scala/org/apache/spark/shuffle/sort/SortShuffleManagerSuite.scala 9 additions, 8 deletions...g/apache/spark/shuffle/sort/SortShuffleManagerSuite.scala
Loading
Please register or sign in to comment