-
- Downloads
[SPARK-14363] Fix executor OOM due to memory leak in the Sorter
## What changes were proposed in this pull request? Fix memory leak in the Sorter. When the UnsafeExternalSorter spills the data to disk, it does not free up the underlying pointer array. As a result, we see a lot of executor OOM and also memory under utilization. This is a regression partially introduced in PR https://github.com/apache/spark/pull/9241 ## How was this patch tested? Tested by running a job and observed around 30% speedup after this change. Author: Sital Kedia <skedia@fb.com> Closes #12285 from sitalkedia/executor_oom.
Showing
- core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java 4 additions, 2 deletions.../org/apache/spark/shuffle/sort/ShuffleExternalSorter.java
- core/src/main/java/org/apache/spark/shuffle/sort/ShuffleInMemorySorter.java 7 additions, 0 deletions.../org/apache/spark/shuffle/sort/ShuffleInMemorySorter.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java 5 additions, 2 deletions...ark/util/collection/unsafe/sort/UnsafeExternalSorter.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java 7 additions, 0 deletions...ark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
Loading
Please register or sign in to comment