Skip to content
Snippets Groups Projects
Commit 1b38f5f2 authored by Patrick Wendell's avatar Patrick Wendell
Browse files

Merge pull request #236 from pwendell/shuffle-docs

Adding disclaimer for shuffle file consolidation
parents e5d5728b b9451acd
No related branches found
No related tags found
No related merge requests found
...@@ -331,7 +331,7 @@ Apart from these, the following properties are also available, and may be useful ...@@ -331,7 +331,7 @@ Apart from these, the following properties are also available, and may be useful
<td>spark.shuffle.consolidateFiles</td> <td>spark.shuffle.consolidateFiles</td>
<td>false</td> <td>false</td>
<td> <td>
If set to "true", consolidates intermediate files created during a shuffle. Creating fewer files can improve filesystem performance if you run shuffles with large numbers of reduce tasks. If set to "true", consolidates intermediate files created during a shuffle. Creating fewer files can improve filesystem performance for shuffles with large numbers of reduce tasks. It is reccomended to set this to "true" when using ext4 or xfs filesystems. On ext3, this option might degrade performance on machines with many (>8) cores due to filesystem limitations.
</td> </td>
</tr> </tr>
<tr> <tr>
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment