Skip to content
Snippets Groups Projects
  • Chris Heller's avatar
    8f50a07d
    [SPARK-2691] [MESOS] Support for Mesos DockerInfo · 8f50a07d
    Chris Heller authored
    This patch adds partial support for running spark on mesos inside of a docker container. Only fine-grained mode is presently supported, and there is no checking done to ensure that the version of libmesos is recent enough to have a DockerInfo structure in the protobuf (other than pinning a mesos version in the pom.xml).
    
    Author: Chris Heller <hellertime@gmail.com>
    
    Closes #3074 from hellertime/SPARK-2691 and squashes the following commits:
    
    d504af6 [Chris Heller] Assist type inference
    f64885d [Chris Heller] Fix errant line length
    17c41c0 [Chris Heller] Base Dockerfile on mesosphere/mesos image
    8aebda4 [Chris Heller] Simplfy Docker image docs
    1ae7f4f [Chris Heller] Style points
    974bd56 [Chris Heller] Convert map to flatMap
    5d8bdf7 [Chris Heller] Factor out the DockerInfo construction.
    7b75a3d [Chris Heller] Align to styleguide
    80108e7 [Chris Heller] Bend to the will of RAT
    ba77056 [Chris Heller] Explicit RAT exclude
    abda5e5 [Chris Heller] Wildcard .rat-excludes
    2f2873c [Chris Heller] Exclude spark-mesos from RAT
    a589a5b [Chris Heller] Add example Dockerfile
    b6825ce [Chris Heller] Remove use of EasyMock
    eae1b86 [Chris Heller] Move properties under 'spark.mesos.'
    c184d00 [Chris Heller] Use map on Option to be consistent with non-coarse code
    fb9501a [Chris Heller] Bumped mesos version to current release
    fa11879 [Chris Heller] Add listenerBus to EasyMock
    882151e [Chris Heller] Changes to scala style
    b22d42d [Chris Heller] Exclude template from RAT
    db536cf [Chris Heller] Remove unneeded mocks
    dea1bd5 [Chris Heller] Force default protocol
    7dac042 [Chris Heller] Add test for DockerInfo
    5456c0c [Chris Heller] Adjust syntax style
    521c194 [Chris Heller] Adjust version info
    6e38f70 [Chris Heller] Document Mesos Docker properties
    29572ab [Chris Heller] Support all DockerInfo fields
    b8c0dea [Chris Heller] Support for mesos DockerInfo in coarse-mode.
    482a9fd [Chris Heller] Support for mesos DockerInfo in fine-grained mode.
    8f50a07d
    History
    [SPARK-2691] [MESOS] Support for Mesos DockerInfo
    Chris Heller authored
    This patch adds partial support for running spark on mesos inside of a docker container. Only fine-grained mode is presently supported, and there is no checking done to ensure that the version of libmesos is recent enough to have a DockerInfo structure in the protobuf (other than pinning a mesos version in the pom.xml).
    
    Author: Chris Heller <hellertime@gmail.com>
    
    Closes #3074 from hellertime/SPARK-2691 and squashes the following commits:
    
    d504af6 [Chris Heller] Assist type inference
    f64885d [Chris Heller] Fix errant line length
    17c41c0 [Chris Heller] Base Dockerfile on mesosphere/mesos image
    8aebda4 [Chris Heller] Simplfy Docker image docs
    1ae7f4f [Chris Heller] Style points
    974bd56 [Chris Heller] Convert map to flatMap
    5d8bdf7 [Chris Heller] Factor out the DockerInfo construction.
    7b75a3d [Chris Heller] Align to styleguide
    80108e7 [Chris Heller] Bend to the will of RAT
    ba77056 [Chris Heller] Explicit RAT exclude
    abda5e5 [Chris Heller] Wildcard .rat-excludes
    2f2873c [Chris Heller] Exclude spark-mesos from RAT
    a589a5b [Chris Heller] Add example Dockerfile
    b6825ce [Chris Heller] Remove use of EasyMock
    eae1b86 [Chris Heller] Move properties under 'spark.mesos.'
    c184d00 [Chris Heller] Use map on Option to be consistent with non-coarse code
    fb9501a [Chris Heller] Bumped mesos version to current release
    fa11879 [Chris Heller] Add listenerBus to EasyMock
    882151e [Chris Heller] Changes to scala style
    b22d42d [Chris Heller] Exclude template from RAT
    db536cf [Chris Heller] Remove unneeded mocks
    dea1bd5 [Chris Heller] Force default protocol
    7dac042 [Chris Heller] Add test for DockerInfo
    5456c0c [Chris Heller] Adjust syntax style
    521c194 [Chris Heller] Adjust version info
    6e38f70 [Chris Heller] Document Mesos Docker properties
    29572ab [Chris Heller] Support all DockerInfo fields
    b8c0dea [Chris Heller] Support for mesos DockerInfo in coarse-mode.
    482a9fd [Chris Heller] Support for mesos DockerInfo in fine-grained mode.
docker.properties.template 202 B