Skip to content
Snippets Groups Projects
  • Kay Ousterhout's avatar
    38702487
    [SPARK-1466] Raise exception if pyspark Gateway process doesn't start. · 38702487
    Kay Ousterhout authored
    If the gateway process fails to start correctly (e.g., because JAVA_HOME isn't set correctly, there's no Spark jar, etc.), right now pyspark fails because of a very difficult-to-understand error, where we try to parse stdout to get the port where Spark started and there's nothing there. This commit properly catches the error and throws an exception that includes the stderr output for much easier debugging.
    
    Thanks to @shivaram and @stogers for helping to fix this issue!
    
    Author: Kay Ousterhout <kayousterhout@gmail.com>
    
    Closes #383 from kayousterhout/pyspark and squashes the following commits:
    
    36dd54b [Kay Ousterhout] [SPARK-1466] Raise exception if Gateway process doesn't start.
    38702487
    History
    [SPARK-1466] Raise exception if pyspark Gateway process doesn't start.
    Kay Ousterhout authored
    If the gateway process fails to start correctly (e.g., because JAVA_HOME isn't set correctly, there's no Spark jar, etc.), right now pyspark fails because of a very difficult-to-understand error, where we try to parse stdout to get the port where Spark started and there's nothing there. This commit properly catches the error and throws an exception that includes the stderr output for much easier debugging.
    
    Thanks to @shivaram and @stogers for helping to fix this issue!
    
    Author: Kay Ousterhout <kayousterhout@gmail.com>
    
    Closes #383 from kayousterhout/pyspark and squashes the following commits:
    
    36dd54b [Kay Ousterhout] [SPARK-1466] Raise exception if Gateway process doesn't start.