Closes #2387

Author: Matthew Farrellee <matt@redhat.com>

Closes #2301 from mattf/SPARK-3425 and squashes the following commits:

20f3c09 [Matthew Farrellee] [SPARK-3425] do not set MaxPermSize for OpenJDK 1.8

As sryza reported, spark-shell doesn't accept any flags.
The root cause is wrong usage of spark-submit in spark-shell and it come to the surface by #1801

Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>
Author: Cheng Lian <lian.cs.zju@gmail.com>

Closes #1715, Closes #1864, and Closes #1861

Closes #1825 from sarutak/SPARK-2894 and squashes the following commits:

47f3510 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-2894
2c899ed [Kousuke Saruta] Removed useless code from java_gateway.py
98287ed [Kousuke Saruta] Removed useless code from java_gateway.py
513ad2e [Kousuke Saruta] Modified util.sh to enable to use option including white spaces
28a374e [Kousuke Saruta] Modified java_gateway.py to recognize arguments
5afc584 [Cheng Lian] Filter out spark-submit options when starting Python gateway
e630d19 [Cheng Lian] Fixing pyspark and spark-shell CLI options

This commit fixes a couple of issues in the merge_spark_pr.py developer script:

- Allow recovery from failed cherry-picks.
- Fix detection of pull requests that have already been merged.

Both of these fixes are useful when backporting changes.

Author: Josh Rosen <joshrosen@apache.org>

Closes #1668 from JoshRosen/pr-script-improvements and squashes the following commits:

ff4f33a [Josh Rosen] Default SPARK_HOME to cwd(); detect missing JIRA credentials.
ed5bc57 [Josh Rosen] Improvements for backporting using merge_spark_pr:

Currently all of the commits with 'X' in them cause person X to
receive e-mails every time someone makes a public fork of Spark.

marmbrus who requested this.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #1239 from pwendell/strip and squashes the following commits:

22e5a97 [Patrick Wendell] Strip '@' symbols when merging pull requests.

Discovered in #992

Author: Patrick Wendell <pwendell@gmail.com>

Closes #1007 from pwendell/hotfix and squashes the following commits:

af90aa0 [Patrick Wendell] HOTFIX: Support empty body in merge script

A minor change to use env default python instead of fixed `/usr/bin/python`.

Author: Xiangrui Meng <meng@databricks.com>

Closes #965 from mengxr/merge-pr-python and squashes the following commits:

1ae0013 [Xiangrui Meng] use env default python in merge_spark_pr.py

Author: Reynold Xin <rxin@apache.org>

Closes #875 from rxin/pep8-dev-scripts and squashes the following commits:

04b084f [Reynold Xin] Made dev Python scripts PEP8 compliant.

Adds an automated hook in the merge script that can close the JIRA,
set the fix versions, and leave a comment on the JIRA indicating the
PR in which it was resolved. This ensures that (a) we always close JIRA's
when issues are merged and (b) there is a link to the pull request in every JIRA.

This requires a python library called `jira-client`. We could look at embedding this
library in our project, but it seemed simple enough to just gracefully disable this
feature if it is not installed. It can be installed with `pip install jira-client`.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #570 from pwendell/jira-pr-merge and squashes the following commits:

3022b96 [Patrick Wendell] SPARK-1648 Support closing JIRA's as part of merge script.

Sometimes if there is a small conflict it's nice to be able to just
manually fix it up rather than have another RTT with the contributor.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #342 from pwendell/merge-conflicts and squashes the following commits:

cdce61a [Patrick Wendell] SPARK-1431: Allow merging conflicting pull requests

Author: Thomas Graves <tgraves@apache.org>

Closes #47 from tgravescs/fix_merge_script and squashes the following commits:

8209ab1 [Thomas Graves] Update dev merge script to use spark.git instead of incubator-spark

The first line of a git commit message is the line that's used with many git
tools as the most concise textual description of that message.  The most
common use that I see is in the short log, which is a one line per commit
log of recent commits.

This commit moves the line

  Merge pull request #%s from %s.

Lower into the message to reserve the first line of the resulting commit for
the much more important pull request title.

http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html

Author: Andrew Ash <andrew@andrewash.com>

Closes #574 from ash211/gh-pr-merge-title and squashes the following commits:

b240823 [Andrew Ash] More merge_message improvements
d2986db [Andrew Ash] Keep GitHub pull request title as commit summary

Fixes bug where merges won't close associated pull request.

Previously we added "Closes #XX" in the title. Github will sometimes
linbreak the title in a way that causes this to not work. This patch
instead adds the line in the body.

This also makes the commit format more concise for merge commits.
We might consider just dropping those in the future.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #569 and squashes the following commits:

732eba1 [Patrick Wendell] Fixes bug where merges won't close associated pull request.

SPARK-1066: Add developer scripts to repository.

These are some developer scripts I've been maintaining in a separate public repo. This patch adds them to the Spark repository so they can evolve here and are clearly accessible to all committers.

I may do some small additional clean-up in this PR, but wanted to put them here in case others want to review. There are a few types of scripts here:

1. A tool to merge pull requests.
2. A script for packaging releases.
3. A script for auditing release candidates.

Author: Patrick Wendell <pwendell@gmail.com>

== Merge branch commits ==

commit 5d5d331d01f6fd59c2eb830f652955119b012173
Author: Patrick Wendell <pwendell@gmail.com>
Date:   Sat Feb 8 22:11:47 2014 -0800

    SPARK-1066: Add developer scripts to repository.