Skip to content
Snippets Groups Projects
  • Prashant Sharma's avatar
    628932b8
    [SPARK-1776] Have Spark's SBT build read dependencies from Maven. · 628932b8
    Prashant Sharma authored
    Patch introduces the new way of working also retaining the existing ways of doing things.
    
    For example build instruction for yarn in maven is
    `mvn -Pyarn -PHadoop2.2 clean package -DskipTests`
    in sbt it can become
    `MAVEN_PROFILES="yarn, hadoop-2.2" sbt/sbt clean assembly`
    Also supports
    `sbt/sbt -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 clean assembly`
    
    Author: Prashant Sharma <prashant.s@imaginea.com>
    Author: Patrick Wendell <pwendell@gmail.com>
    
    Closes #772 from ScrapCodes/sbt-maven and squashes the following commits:
    
    a8ac951 [Prashant Sharma] Updated sbt version.
    62b09bb [Prashant Sharma] Improvements.
    fa6221d [Prashant Sharma] Excluding sql from mima
    4b8875e [Prashant Sharma] Sbt assembly no longer builds tools by default.
    72651ca [Prashant Sharma] Addresses code reivew comments.
    acab73d [Prashant Sharma] Revert "Small fix to run-examples script."
    ac4312c [Prashant Sharma] Revert "minor fix"
    6af91ac [Prashant Sharma] Ported oldDeps back. + fixes issues with prev commit.
    65cf06c [Prashant Sharma] Servelet API jars mess up with the other servlet jars on the class path.
    446768e [Prashant Sharma] minor fix
    89b9777 [Prashant Sharma] Merge conflicts
    d0a02f2 [Prashant Sharma] Bumped up pom versions, Since the build now depends on pom it is better updated there. + general cleanups.
    dccc8ac [Prashant Sharma] updated mima to check against 1.0
    a49c61b [Prashant Sharma] Fix for tools jar
    a2f5ae1 [Prashant Sharma] Fixes a bug in dependencies.
    cf88758 [Prashant Sharma] cleanup
    9439ea3 [Prashant Sharma] Small fix to run-examples script.
    96cea1f [Prashant Sharma] SPARK-1776 Have Spark's SBT build read dependencies from Maven.
    36efa62 [Patrick Wendell] Set project name in pom files and added eclipse/intellij plugins.
    4973dbd [Patrick Wendell] Example build using pom reader.
    628932b8
    History
    [SPARK-1776] Have Spark's SBT build read dependencies from Maven.
    Prashant Sharma authored
    Patch introduces the new way of working also retaining the existing ways of doing things.
    
    For example build instruction for yarn in maven is
    `mvn -Pyarn -PHadoop2.2 clean package -DskipTests`
    in sbt it can become
    `MAVEN_PROFILES="yarn, hadoop-2.2" sbt/sbt clean assembly`
    Also supports
    `sbt/sbt -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 clean assembly`
    
    Author: Prashant Sharma <prashant.s@imaginea.com>
    Author: Patrick Wendell <pwendell@gmail.com>
    
    Closes #772 from ScrapCodes/sbt-maven and squashes the following commits:
    
    a8ac951 [Prashant Sharma] Updated sbt version.
    62b09bb [Prashant Sharma] Improvements.
    fa6221d [Prashant Sharma] Excluding sql from mima
    4b8875e [Prashant Sharma] Sbt assembly no longer builds tools by default.
    72651ca [Prashant Sharma] Addresses code reivew comments.
    acab73d [Prashant Sharma] Revert "Small fix to run-examples script."
    ac4312c [Prashant Sharma] Revert "minor fix"
    6af91ac [Prashant Sharma] Ported oldDeps back. + fixes issues with prev commit.
    65cf06c [Prashant Sharma] Servelet API jars mess up with the other servlet jars on the class path.
    446768e [Prashant Sharma] minor fix
    89b9777 [Prashant Sharma] Merge conflicts
    d0a02f2 [Prashant Sharma] Bumped up pom versions, Since the build now depends on pom it is better updated there. + general cleanups.
    dccc8ac [Prashant Sharma] updated mima to check against 1.0
    a49c61b [Prashant Sharma] Fix for tools jar
    a2f5ae1 [Prashant Sharma] Fixes a bug in dependencies.
    cf88758 [Prashant Sharma] cleanup
    9439ea3 [Prashant Sharma] Small fix to run-examples script.
    96cea1f [Prashant Sharma] SPARK-1776 Have Spark's SBT build read dependencies from Maven.
    36efa62 [Patrick Wendell] Set project name in pom files and added eclipse/intellij plugins.
    4973dbd [Patrick Wendell] Example build using pom reader.
spark-class 5.05 KiB
#!/usr/bin/env bash

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

cygwin=false
case "`uname`" in
    CYGWIN*) cygwin=true;;
esac

SCALA_VERSION=2.10

# Figure out where Spark is installed
FWDIR="$(cd `dirname $0`/..; pwd)"

# Export this as SPARK_HOME
export SPARK_HOME="$FWDIR"

. $FWDIR/bin/load-spark-env.sh

if [ -z "$1" ]; then
  echo "Usage: spark-class <class> [<args>]" 1>&2
  exit 1
fi

if [ -n "$SPARK_MEM" ]; then
  echo -e "Warning: SPARK_MEM is deprecated, please use a more specific config option" 1>&2
  echo -e "(e.g., spark.executor.memory or SPARK_DRIVER_MEMORY)." 1>&2
fi

# Use SPARK_MEM or 512m as the default memory, to be overridden by specific options
DEFAULT_MEM=${SPARK_MEM:-512m}

SPARK_DAEMON_JAVA_OPTS="$SPARK_DAEMON_JAVA_OPTS -Dspark.akka.logLifecycleEvents=true"

# Add java opts and memory settings for master, worker, history server, executors, and repl.
case "$1" in
  # Master, Worker, and HistoryServer use SPARK_DAEMON_JAVA_OPTS (and specific opts) + SPARK_DAEMON_MEMORY.
  'org.apache.spark.deploy.master.Master')
    OUR_JAVA_OPTS="$SPARK_DAEMON_JAVA_OPTS $SPARK_MASTER_OPTS"
    OUR_JAVA_MEM=${SPARK_DAEMON_MEMORY:-$DEFAULT_MEM}
    ;;
  'org.apache.spark.deploy.worker.Worker')
    OUR_JAVA_OPTS="$SPARK_DAEMON_JAVA_OPTS $SPARK_WORKER_OPTS"
    OUR_JAVA_MEM=${SPARK_DAEMON_MEMORY:-$DEFAULT_MEM}
    ;;
  'org.apache.spark.deploy.history.HistoryServer')
    OUR_JAVA_OPTS="$SPARK_DAEMON_JAVA_OPTS $SPARK_HISTORY_OPTS"
    OUR_JAVA_MEM=${SPARK_DAEMON_MEMORY:-$DEFAULT_MEM}
    ;;

  # Executors use SPARK_JAVA_OPTS + SPARK_EXECUTOR_MEMORY.
  'org.apache.spark.executor.CoarseGrainedExecutorBackend')
    OUR_JAVA_OPTS="$SPARK_JAVA_OPTS $SPARK_EXECUTOR_OPTS"
    OUR_JAVA_MEM=${SPARK_EXECUTOR_MEMORY:-$DEFAULT_MEM}
    ;;
  'org.apache.spark.executor.MesosExecutorBackend')
    OUR_JAVA_OPTS="$SPARK_JAVA_OPTS $SPARK_EXECUTOR_OPTS"
    OUR_JAVA_MEM=${SPARK_EXECUTOR_MEMORY:-$DEFAULT_MEM}
    ;;

  # Spark submit uses SPARK_SUBMIT_OPTS and SPARK_JAVA_OPTS
    'org.apache.spark.deploy.SparkSubmit')
    OUR_JAVA_OPTS="$SPARK_JAVA_OPTS $SPARK_SUBMIT_OPTS \
      -Djava.library.path=$SPARK_SUBMIT_LIBRARY_PATH"
    OUR_JAVA_MEM=${SPARK_DRIVER_MEMORY:-$DEFAULT_MEM}
    ;;

  *)
    OUR_JAVA_OPTS="$SPARK_JAVA_OPTS"
    OUR_JAVA_MEM=${SPARK_DRIVER_MEMORY:-$DEFAULT_MEM}
    ;;
esac

# Find the java binary
if [ -n "${JAVA_HOME}" ]; then
  RUNNER="${JAVA_HOME}/bin/java"
else
  if [ `command -v java` ]; then
    RUNNER="java"
  else
    echo "JAVA_HOME is not set" >&2
    exit 1
  fi
fi

# Set JAVA_OPTS to be able to load native libraries and to set heap size
JAVA_OPTS="-XX:MaxPermSize=128m $OUR_JAVA_OPTS"
JAVA_OPTS="$JAVA_OPTS -Xms$OUR_JAVA_MEM -Xmx$OUR_JAVA_MEM"
# Load extra JAVA_OPTS from conf/java-opts, if it exists
if [ -e "$FWDIR/conf/java-opts" ] ; then
  JAVA_OPTS="$JAVA_OPTS `cat $FWDIR/conf/java-opts`"
fi
export JAVA_OPTS
# Attention: when changing the way the JAVA_OPTS are assembled, the change must be reflected in CommandUtils.scala!

TOOLS_DIR="$FWDIR"/tools
SPARK_TOOLS_JAR=""
if [ -e "$TOOLS_DIR"/target/scala-$SCALA_VERSION/spark-tools*[0-9Tg].jar ]; then
  # Use the JAR from the SBT build
  export SPARK_TOOLS_JAR=`ls "$TOOLS_DIR"/target/scala-$SCALA_VERSION/spark-tools*[0-9Tg].jar`
fi
if [ -e "$TOOLS_DIR"/target/spark-tools*[0-9Tg].jar ]; then
  # Use the JAR from the Maven build
  # TODO: this also needs to become an assembly!
  export SPARK_TOOLS_JAR=`ls "$TOOLS_DIR"/target/spark-tools*[0-9Tg].jar`
fi

# Compute classpath using external script
classpath_output=$($FWDIR/bin/compute-classpath.sh)
if [[ "$?" != "0" ]]; then
  echo "$classpath_output"
  exit 1
else
  CLASSPATH=$classpath_output
fi

if [[ "$1" =~ org.apache.spark.tools.* ]]; then
  if test -z "$SPARK_TOOLS_JAR"; then
    echo "Failed to find Spark Tools Jar in $FWDIR/tools/target/scala-$SCALA_VERSION/" 1>&2
    echo "You need to build spark before running $1." 1>&2
    exit 1
  fi
  CLASSPATH="$CLASSPATH:$SPARK_TOOLS_JAR"
fi

if $cygwin; then
  CLASSPATH=`cygpath -wp $CLASSPATH`
  if [ "$1" == "org.apache.spark.tools.JavaAPICompletenessChecker" ]; then
    export SPARK_TOOLS_JAR=`cygpath -w $SPARK_TOOLS_JAR`
  fi
fi
export CLASSPATH

if [ "$SPARK_PRINT_LAUNCH_COMMAND" == "1" ]; then
  echo -n "Spark Command: " 1>&2
  echo "$RUNNER" -cp "$CLASSPATH" $JAVA_OPTS "$@" 1>&2
  echo -e "========================================\n" 1>&2
fi

exec "$RUNNER" -cp "$CLASSPATH" $JAVA_OPTS "$@"