Skip to content
Snippets Groups Projects
Commit f38fab97 authored by Rob O'Dwyer's avatar Rob O'Dwyer Committed by Matei Zaharia
Browse files

SPARK-3265 Allow using custom ipython executable with pyspark

Although you can make pyspark use ipython with `IPYTHON=1`, and also change the python executable with `PYSPARK_PYTHON=...`, you can't use both at the same time because it hardcodes the default ipython script.

This makes it use the `PYSPARK_PYTHON` variable if present and fall back to default python, similarly to how the default python executable is handled.

So you can use a custom ipython like so:
`PYSPARK_PYTHON=./anaconda/bin/ipython IPYTHON_OPTS="notebook" pyspark`

Author: Rob O'Dwyer <odwyerrob@gmail.com>

Closes #2167 from robbles/patch-1 and squashes the following commits:

d98e8a9 [Rob O'Dwyer] Allow using custom ipython executable with pyspark
parent b86277c1
No related branches found
No related tags found
No related merge requests found
...@@ -104,7 +104,7 @@ if [[ "$1" =~ \.py$ ]]; then ...@@ -104,7 +104,7 @@ if [[ "$1" =~ \.py$ ]]; then
else else
# Only use ipython if no command line arguments were provided [SPARK-1134] # Only use ipython if no command line arguments were provided [SPARK-1134]
if [[ "$IPYTHON" = "1" ]]; then if [[ "$IPYTHON" = "1" ]]; then
exec ipython $IPYTHON_OPTS exec ${PYSPARK_PYTHON:-ipython} $IPYTHON_OPTS
else else
exec "$PYSPARK_PYTHON" exec "$PYSPARK_PYTHON"
fi fi
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment