-
- Downloads
[SPARK-9452] [SQL] Support records larger than page size in UnsafeExternalSorter
This patch extends UnsafeExternalSorter to support records larger than the page size. The basic strategy is the same as in #7762: store large records in their own overflow pages. Author: Josh Rosen <joshrosen@databricks.com> Closes #7891 from JoshRosen/large-records-in-sql-sorter and squashes the following commits: 967580b [Josh Rosen] Merge remote-tracking branch 'origin/master' into large-records-in-sql-sorter 948c344 [Josh Rosen] Add large records tests for KV sorter. 3c17288 [Josh Rosen] Combine memory and disk cleanup into general cleanupResources() method 380f217 [Josh Rosen] Merge remote-tracking branch 'origin/master' into large-records-in-sql-sorter 27eafa0 [Josh Rosen] Fix page size in PackedRecordPointerSuite a49baef [Josh Rosen] Address initial round of review comments 3edb931 [Josh Rosen] Remove accidentally-committed debug statements. 2b164e2 [Josh Rosen] Support large records in UnsafeExternalSorter.
Showing
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java 121 additions, 52 deletions...ark/util/collection/unsafe/sort/UnsafeExternalSorter.java
- core/src/test/java/org/apache/spark/shuffle/unsafe/PackedRecordPointerSuite.java 4 additions, 4 deletions...apache/spark/shuffle/unsafe/PackedRecordPointerSuite.java
- core/src/test/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorterSuite.java 98 additions, 31 deletions...til/collection/unsafe/sort/UnsafeExternalSorterSuite.java
- sql/catalyst/src/main/java/org/apache/spark/sql/execution/UnsafeExternalRowSorter.java 1 addition, 1 deletion...g/apache/spark/sql/execution/UnsafeExternalRowSorter.java
- sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeKVExternalSorter.java 9 additions, 2 deletions...rg/apache/spark/sql/execution/UnsafeKVExternalSorter.java
- sql/core/src/test/scala/org/apache/spark/sql/execution/UnsafeKVExternalSorterSuite.scala 133 additions, 77 deletions...che/spark/sql/execution/UnsafeKVExternalSorterSuite.scala
- unsafe/src/main/java/org/apache/spark/unsafe/memory/HeapMemoryAllocator.java 3 additions, 0 deletions...a/org/apache/spark/unsafe/memory/HeapMemoryAllocator.java
- unsafe/src/main/java/org/apache/spark/unsafe/memory/UnsafeMemoryAllocator.java 3 additions, 0 deletions...org/apache/spark/unsafe/memory/UnsafeMemoryAllocator.java
Loading
Please register or sign in to comment