Skip to content
Snippets Groups Projects
  • Josh Rosen's avatar
    2117eea7
    [SPARK-10710] Remove ability to disable spilling in core and SQL · 2117eea7
    Josh Rosen authored
    It does not make much sense to set `spark.shuffle.spill` or `spark.sql.planner.externalSort` to false: I believe that these configurations were initially added as "escape hatches" to guard against bugs in the external operators, but these operators are now mature and well-tested. In addition, these configurations are not handled in a consistent way anymore: SQL's Tungsten codepath ignores these configurations and will continue to use spilling operators. Similarly, Spark Core's `tungsten-sort` shuffle manager does not respect `spark.shuffle.spill=false`.
    
    This pull request removes these configurations, adds warnings at the appropriate places, and deletes a large amount of code which was only used in code paths that did not support spilling.
    
    Author: Josh Rosen <joshrosen@databricks.com>
    
    Closes #8831 from JoshRosen/remove-ability-to-disable-spilling.
    2117eea7
    History
    [SPARK-10710] Remove ability to disable spilling in core and SQL
    Josh Rosen authored
    It does not make much sense to set `spark.shuffle.spill` or `spark.sql.planner.externalSort` to false: I believe that these configurations were initially added as "escape hatches" to guard against bugs in the external operators, but these operators are now mature and well-tested. In addition, these configurations are not handled in a consistent way anymore: SQL's Tungsten codepath ignores these configurations and will continue to use spilling operators. Similarly, Spark Core's `tungsten-sort` shuffle manager does not respect `spark.shuffle.spill=false`.
    
    This pull request removes these configurations, adds warnings at the appropriate places, and deletes a large amount of code which was only used in code paths that did not support spilling.
    
    Author: Josh Rosen <joshrosen@databricks.com>
    
    Closes #8831 from JoshRosen/remove-ability-to-disable-spilling.