-
- Downloads
Modified shuffle to limit the maximum outstanding data size in bytes,
instead of the maximum number of outstanding fetches. This should make it faster when there are many small map output files, as well as more robust to overallocating memory on large map outputs.
Showing
- core/src/main/scala/spark/BlockStoreShuffleFetcher.scala 7 additions, 7 deletionscore/src/main/scala/spark/BlockStoreShuffleFetcher.scala
- core/src/main/scala/spark/storage/BlockManager.scala 83 additions, 50 deletionscore/src/main/scala/spark/storage/BlockManager.scala
- core/src/test/scala/spark/DistributedSuite.scala 17 additions, 4 deletionscore/src/test/scala/spark/DistributedSuite.scala
- core/src/test/scala/spark/MapOutputTrackerSuite.scala 2 additions, 0 deletionscore/src/test/scala/spark/MapOutputTrackerSuite.scala
- docs/configuration.md 5 additions, 3 deletionsdocs/configuration.md
Loading
Please register or sign in to comment