Skip to content
Snippets Groups Projects
  • Iulian Dragos's avatar
    8aab94d8
    [SPARK-4286] Add an external shuffle service that can be run as a daemon. · 8aab94d8
    Iulian Dragos authored
    This allows Mesos deployments to use the shuffle service (and implicitly dynamic allocation). It does so by adding a new "main" class and two corresponding scripts in `sbin`:
    
    - `sbin/start-shuffle-service.sh`
    - `sbin/stop-shuffle-service.sh`
    
    Specific options can be passed in `SPARK_SHUFFLE_OPTS`.
    
    This is picking up work from #3861 /cc tnachen
    
    Author: Iulian Dragos <jaguarul@gmail.com>
    
    Closes #4990 from dragos/feature/external-shuffle-service and squashes the following commits:
    
    6c2b148 [Iulian Dragos] Import order and wrong name fixup.
    07804ad [Iulian Dragos] Moved ExternalShuffleService to the `deploy` package + other minor tweaks.
    4dc1f91 [Iulian Dragos] Reviewer’s comments:
    8145429 [Iulian Dragos] Add an external shuffle service that can be run as a daemon.
    8aab94d8
    History
    [SPARK-4286] Add an external shuffle service that can be run as a daemon.
    Iulian Dragos authored
    This allows Mesos deployments to use the shuffle service (and implicitly dynamic allocation). It does so by adding a new "main" class and two corresponding scripts in `sbin`:
    
    - `sbin/start-shuffle-service.sh`
    - `sbin/stop-shuffle-service.sh`
    
    Specific options can be passed in `SPARK_SHUFFLE_OPTS`.
    
    This is picking up work from #3861 /cc tnachen
    
    Author: Iulian Dragos <jaguarul@gmail.com>
    
    Closes #4990 from dragos/feature/external-shuffle-service and squashes the following commits:
    
    6c2b148 [Iulian Dragos] Import order and wrong name fixup.
    07804ad [Iulian Dragos] Moved ExternalShuffleService to the `deploy` package + other minor tweaks.
    4dc1f91 [Iulian Dragos] Reviewer’s comments:
    8145429 [Iulian Dragos] Add an external shuffle service that can be run as a daemon.
auth NaN GiB