Skip to content
Snippets Groups Projects
  • Ilya Ganelin's avatar
    3697232b
    [SPARK-3071] Increase default driver memory · 3697232b
    Ilya Ganelin authored
    I've updated default values in comments, documentation, and in the command line builder to be 1g based on comments in the JIRA. I've also updated most usages to point at a single variable defined in the Utils.scala and JavaUtils.java files. This wasn't possible in all cases (R, shell scripts etc.) but usage in most code is now pointing at the same place.
    
    Please let me know if I've missed anything.
    
    Will the spark-shell use the value within the command line builder during instantiation?
    
    Author: Ilya Ganelin <ilya.ganelin@capitalone.com>
    
    Closes #7132 from ilganeli/SPARK-3071 and squashes the following commits:
    
    4074164 [Ilya Ganelin] String fix
    271610b [Ilya Ganelin] Merge branch 'SPARK-3071' of github.com:ilganeli/spark into SPARK-3071
    273b6e9 [Ilya Ganelin] Test fix
    fd67721 [Ilya Ganelin] Update JavaUtils.java
    26cc177 [Ilya Ganelin] test fix
    e5db35d [Ilya Ganelin] Fixed test failure
    39732a1 [Ilya Ganelin] merge fix
    a6f7deb [Ilya Ganelin] Created default value for DRIVER MEM in Utils that's now used in almost all locations instead of setting manually in each
    09ad698 [Ilya Ganelin] Update SubmitRestProtocolSuite.scala
    19b6f25 [Ilya Ganelin] Missed one doc update
    2698a3d [Ilya Ganelin] Updated default value for driver memory
    3697232b
    History
    [SPARK-3071] Increase default driver memory
    Ilya Ganelin authored
    I've updated default values in comments, documentation, and in the command line builder to be 1g based on comments in the JIRA. I've also updated most usages to point at a single variable defined in the Utils.scala and JavaUtils.java files. This wasn't possible in all cases (R, shell scripts etc.) but usage in most code is now pointing at the same place.
    
    Please let me know if I've missed anything.
    
    Will the spark-shell use the value within the command line builder during instantiation?
    
    Author: Ilya Ganelin <ilya.ganelin@capitalone.com>
    
    Closes #7132 from ilganeli/SPARK-3071 and squashes the following commits:
    
    4074164 [Ilya Ganelin] String fix
    271610b [Ilya Ganelin] Merge branch 'SPARK-3071' of github.com:ilganeli/spark into SPARK-3071
    273b6e9 [Ilya Ganelin] Test fix
    fd67721 [Ilya Ganelin] Update JavaUtils.java
    26cc177 [Ilya Ganelin] test fix
    e5db35d [Ilya Ganelin] Fixed test failure
    39732a1 [Ilya Ganelin] merge fix
    a6f7deb [Ilya Ganelin] Created default value for DRIVER MEM in Utils that's now used in almost all locations instead of setting manually in each
    09ad698 [Ilya Ganelin] Update SubmitRestProtocolSuite.scala
    19b6f25 [Ilya Ganelin] Missed one doc update
    2698a3d [Ilya Ganelin] Updated default value for driver memory
spark-env.sh.template 3.24 KiB