Skip to content
Snippets Groups Projects
  • Andrew Or's avatar
    2ffd1eaf
    [SPARK-1753 / 1773 / 1814] Update outdated docs for spark-submit, YARN, standalone etc. · 2ffd1eaf
    Andrew Or authored
    YARN
    - SparkPi was updated to not take in master as an argument; we should update the docs to reflect that.
    - The default YARN build guide should be in maven, not sbt.
    - This PR also adds a paragraph on steps to debug a YARN application.
    
    Standalone
    - Emphasize spark-submit more. Right now it's one small paragraph preceding the legacy way of launching through `org.apache.spark.deploy.Client`.
    - The way we set configurations / environment variables according to the old docs is outdated. This needs to reflect changes introduced by the Spark configuration changes we made.
    
    In general, this PR also adds a little more documentation on the new spark-shell, spark-submit, spark-defaults.conf etc here and there.
    
    Author: Andrew Or <andrewor14@gmail.com>
    
    Closes #701 from andrewor14/yarn-docs and squashes the following commits:
    
    e2c2312 [Andrew Or] Merge in changes in #752 (SPARK-1814)
    25cfe7b [Andrew Or] Merge in the warning from SPARK-1753
    a8c39c5 [Andrew Or] Minor changes
    336bbd9 [Andrew Or] Tabs -> spaces
    4d9d8f7 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs
    041017a [Andrew Or] Abstract Spark submit documentation to cluster-overview.html
    3cc0649 [Andrew Or] Detail how to set configurations + remove legacy instructions
    5b7140a [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs
    85a51fc [Andrew Or] Update run-example, spark-shell, configuration etc.
    c10e8c7 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs
    381fe32 [Andrew Or] Update docs for standalone mode
    757c184 [Andrew Or] Add a note about the requirements for the debugging trick
    f8ca990 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs
    924f04c [Andrew Or] Revert addition of --deploy-mode
    d5fe17b [Andrew Or] Update the YARN docs
    2ffd1eaf
    History
    [SPARK-1753 / 1773 / 1814] Update outdated docs for spark-submit, YARN, standalone etc.
    Andrew Or authored
    YARN
    - SparkPi was updated to not take in master as an argument; we should update the docs to reflect that.
    - The default YARN build guide should be in maven, not sbt.
    - This PR also adds a paragraph on steps to debug a YARN application.
    
    Standalone
    - Emphasize spark-submit more. Right now it's one small paragraph preceding the legacy way of launching through `org.apache.spark.deploy.Client`.
    - The way we set configurations / environment variables according to the old docs is outdated. This needs to reflect changes introduced by the Spark configuration changes we made.
    
    In general, this PR also adds a little more documentation on the new spark-shell, spark-submit, spark-defaults.conf etc here and there.
    
    Author: Andrew Or <andrewor14@gmail.com>
    
    Closes #701 from andrewor14/yarn-docs and squashes the following commits:
    
    e2c2312 [Andrew Or] Merge in changes in #752 (SPARK-1814)
    25cfe7b [Andrew Or] Merge in the warning from SPARK-1753
    a8c39c5 [Andrew Or] Minor changes
    336bbd9 [Andrew Or] Tabs -> spaces
    4d9d8f7 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs
    041017a [Andrew Or] Abstract Spark submit documentation to cluster-overview.html
    3cc0649 [Andrew Or] Detail how to set configurations + remove legacy instructions
    5b7140a [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs
    85a51fc [Andrew Or] Update run-example, spark-shell, configuration etc.
    c10e8c7 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs
    381fe32 [Andrew Or] Update docs for standalone mode
    757c184 [Andrew Or] Add a note about the requirements for the debugging trick
    f8ca990 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs
    924f04c [Andrew Or] Revert addition of --deploy-mode
    d5fe17b [Andrew Or] Update the YARN docs
building-with-maven.md 6.35 KiB
layout: global
title: Building Spark with Maven
  • This will become a table of contents (this text will be scraped). {:toc}

Building Spark using Maven requires Maven 3.0.4 or newer and Java 1.6 or newer.

Setting up Maven's Memory Usage

You'll need to configure Maven to use more memory than usual by setting MAVEN_OPTS. We recommend the following settings:

export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"

If you don't run this, you may see errors like the following:

[INFO] Compiling 203 Scala sources and 9 Java sources to /Users/me/Development/spark/core/target/scala-{{site.SCALA_BINARY_VERSION}}/classes...
[ERROR] PermGen space -> [Help 1]

[INFO] Compiling 203 Scala sources and 9 Java sources to /Users/me/Development/spark/core/target/scala-{{site.SCALA_BINARY_VERSION}}/classes...
[ERROR] Java heap space -> [Help 1]

You can fix this by setting the MAVEN_OPTS variable as discussed before.

Note: For Java 1.8 and above this step is not required.

Specifying the Hadoop version

Because HDFS is not protocol-compatible across versions, if you want to read from HDFS, you'll need to build Spark against the specific HDFS version in your environment. You can do this through the "hadoop.version" property. If unset, Spark will build against Hadoop 1.0.4 by default. Note that certain build profiles are required for particular Hadoop versions:

Hadoop version Profile required
0.23.x hadoop-0.23
1.x to 2.1.x (none)
2.2.x hadoop-2.2
2.3.x hadoop-2.3
2.4.x hadoop-2.4

For Apache Hadoop versions 1.x, Cloudera CDH "mr1" distributions, and other Hadoop versions without YARN, use:

# Apache Hadoop 1.2.1
$ mvn -Dhadoop.version=1.2.1 -DskipTests clean package

# Cloudera CDH 4.2.0 with MapReduce v1
$ mvn -Dhadoop.version=2.0.0-mr1-cdh4.2.0 -DskipTests clean package

# Apache Hadoop 0.23.x
$ mvn -Phadoop-0.23 -Dhadoop.version=0.23.7 -DskipTests clean package

For Apache Hadoop 2.x, 0.23.x, Cloudera CDH, and other Hadoop versions with YARN, you can enable the "yarn-alpha" or "yarn" profile and optionally set the "yarn.version" property if it is different from "hadoop.version". The additional build profile required depends on the YARN version:

YARN version Profile required
0.23.x to 2.1.x yarn-alpha
2.2.x and later yarn

Examples:

# Apache Hadoop 2.0.5-alpha
$ mvn -Pyarn-alpha -Dhadoop.version=2.0.5-alpha -DskipTests clean package

# Cloudera CDH 4.2.0
$ mvn -Pyarn-alpha -Dhadoop.version=2.0.0-cdh4.2.0 -DskipTests clean package

# Apache Hadoop 0.23.x
$ mvn -Pyarn-alpha -Phadoop-0.23 -Dhadoop.version=0.23.7 -DskipTests clean package

# Apache Hadoop 2.2.X
$ mvn -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 -DskipTests clean package

# Apache Hadoop 2.3.X
$ mvn -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -DskipTests clean package

# Apache Hadoop 2.4.X
$ mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package

# Different versions of HDFS and YARN.
$ mvn -Pyarn-alpha -Phadoop-2.3 -Dhadoop.version=2.3.0 -Dyarn.version=0.23.7 -DskipTests clean package

Spark Tests in Maven