-
- Downloads
[SPARK-9700] Pick default page size more intelligently.
Previously, we use 64MB as the default page size, which was way too big for a lot of Spark applications (especially for single node). This patch changes it so that the default page size, if unset by the user, is determined by the number of cores available and the total execution memory available. Author: Reynold Xin <rxin@databricks.com> Closes #8012 from rxin/pagesize and squashes the following commits: 16f4756 [Reynold Xin] Fixed failing test. 5afd570 [Reynold Xin] private... 0d5fb98 [Reynold Xin] Update default value. 674a6cd [Reynold Xin] Address review feedback. dc00e05 [Reynold Xin] Merge with master. 73ebdb6 [Reynold Xin] [SPARK-9700] Pick default page size more intelligently.
Showing
- R/run-tests.sh 1 addition, 1 deletionR/run-tests.sh
- core/src/main/java/org/apache/spark/shuffle/unsafe/UnsafeShuffleExternalSorter.java 1 addition, 2 deletions...che/spark/shuffle/unsafe/UnsafeShuffleExternalSorter.java
- core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java 1 addition, 7 deletions...ain/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java 0 additions, 1 deletion...ark/util/collection/unsafe/sort/UnsafeExternalSorter.java
- core/src/main/scala/org/apache/spark/SparkConf.scala 7 additions, 0 deletionscore/src/main/scala/org/apache/spark/SparkConf.scala
- core/src/main/scala/org/apache/spark/SparkContext.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/SparkEnv.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/SparkEnv.scala
- core/src/main/scala/org/apache/spark/shuffle/ShuffleMemoryManager.scala 49 additions, 4 deletions...scala/org/apache/spark/shuffle/ShuffleMemoryManager.scala
- core/src/test/java/org/apache/spark/shuffle/unsafe/UnsafeShuffleWriterSuite.java 3 additions, 2 deletions...apache/spark/shuffle/unsafe/UnsafeShuffleWriterSuite.java
- core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java 3 additions, 3 deletions...apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
- core/src/test/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorterSuite.java 2 additions, 2 deletions...til/collection/unsafe/sort/UnsafeExternalSorterSuite.java
- core/src/test/scala/org/apache/spark/shuffle/ShuffleMemoryManagerSuite.scala 7 additions, 7 deletions.../org/apache/spark/shuffle/ShuffleMemoryManagerSuite.scala
- python/pyspark/java_gateway.py 0 additions, 1 deletionpython/pyspark/java_gateway.py
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala 1 addition, 1 deletion...sql/execution/aggregate/TungstenAggregationIterator.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala 7 additions, 9 deletions...org/apache/spark/sql/execution/joins/HashedRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/sort.scala 2 additions, 2 deletions.../src/main/scala/org/apache/spark/sql/execution/sort.scala
- sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTypesConverter.scala 0 additions, 0 deletions.../org/apache/spark/sql/parquet/ParquetTypesConverter.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/TestShuffleMemoryManager.scala 1 addition, 1 deletion...apache/spark/sql/execution/TestShuffleMemoryManager.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/test/TestHive.scala 0 additions, 1 deletion.../main/scala/org/apache/spark/sql/hive/test/TestHive.scala
- unsafe/src/main/java/org/apache/spark/unsafe/array/ByteArrayMethods.java 6 additions, 0 deletions.../java/org/apache/spark/unsafe/array/ByteArrayMethods.java
Loading
Please register or sign in to comment