-
- Downloads
[SPARK-19839][CORE] release longArray in BytesToBytesMap
## What changes were proposed in this pull request? When BytesToBytesMap spills, its longArray should be released. Otherwise, it may not released until the task complete. This array may take a significant amount of memory, which cannot be used by later operator, such as UnsafeShuffleExternalSorter, resulting in more frequent spill in sorter. This patch release the array as destructive iterator will not use this array anymore. ## How was this patch tested? Manual test in production Author: Zhan Zhang <zhanzhang@fb.com> Closes #17180 from zhzhan/memory.
Showing
- core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java 5 additions, 0 deletions...ain/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
- sql/core/src/test/scala/org/apache/spark/sql/execution/UnsafeFixedWidthAggregationMapSuite.scala 3 additions, 2 deletions...k/sql/execution/UnsafeFixedWidthAggregationMapSuite.scala
Please register or sign in to comment