-
- Downloads
[SPARK-21175] Reject OpenBlocks when memory shortage on shuffle service.
## What changes were proposed in this pull request? A shuffle service can serves blocks from multiple apps/tasks. Thus the shuffle service can suffers high memory usage when lots of shuffle-reads happen at the same time. In my cluster, OOM always happens on shuffle service. Analyzing heap dump, memory cost by Netty(ChannelOutboundBufferEntry) can be up to 2~3G. It might make sense to reject "open blocks" request when memory usage is high on shuffle service. https://github.com/apache/spark/commit/93dd0c518d040155b04e5ab258c5835aec7776fc and https://github.com/apache/spark/commit/85c6ce61930490e2247fb4b0e22dfebbb8b6a1ee tried to alleviate the memory pressure on shuffle service but cannot solve the root cause. This pr proposes to control currency of shuffle read. ## How was this patch tested? Added unit test. Author: jinxing <jinxing6042@126.com> Closes #18388 from jinxing64/SPARK-21175.
Showing
- common/network-common/src/main/java/org/apache/spark/network/TransportContext.java 1 addition, 1 deletion.../main/java/org/apache/spark/network/TransportContext.java
- common/network-common/src/main/java/org/apache/spark/network/server/OneForOneStreamManager.java 54 additions, 6 deletions...g/apache/spark/network/server/OneForOneStreamManager.java
- common/network-common/src/main/java/org/apache/spark/network/server/StreamManager.java 27 additions, 0 deletions...n/java/org/apache/spark/network/server/StreamManager.java
- common/network-common/src/main/java/org/apache/spark/network/server/TransportRequestHandler.java 36 additions, 6 deletions.../apache/spark/network/server/TransportRequestHandler.java
- common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java 6 additions, 0 deletions...ain/java/org/apache/spark/network/util/TransportConf.java
- common/network-common/src/test/java/org/apache/spark/network/TransportRequestHandlerSuite.java 134 additions, 0 deletions...rg/apache/spark/network/TransportRequestHandlerSuite.java
- docs/configuration.md 7 additions, 0 deletionsdocs/configuration.md
Loading
Please register or sign in to comment