Skip to content
Snippets Groups Projects
  • Andrew Or's avatar
    ba3c730e
    [SPARK-3140] Clarify confusing PySpark exception message · ba3c730e
    Andrew Or authored
    We read the py4j port from the stdout of the `bin/spark-submit` subprocess. If there is interference in stdout (e.g. a random echo in `spark-submit`), we throw an exception with a warning message. We do not, however, distinguish between this case from the case where no stdout is produced at all.
    
    I wasted a non-trivial amount of time being baffled by this exception in search of places where I print random whitespace (in vain, of course). A clearer exception message that distinguishes between these cases will prevent similar headaches that I have gone through.
    
    Author: Andrew Or <andrewor14@gmail.com>
    
    Closes #2067 from andrewor14/python-exception and squashes the following commits:
    
    742f823 [Andrew Or] Further clarify warning messages
    e96a7a0 [Andrew Or] Distinguish between unexpected output and no output at all
    ba3c730e
    History
    [SPARK-3140] Clarify confusing PySpark exception message
    Andrew Or authored
    We read the py4j port from the stdout of the `bin/spark-submit` subprocess. If there is interference in stdout (e.g. a random echo in `spark-submit`), we throw an exception with a warning message. We do not, however, distinguish between this case from the case where no stdout is produced at all.
    
    I wasted a non-trivial amount of time being baffled by this exception in search of places where I print random whitespace (in vain, of course). A clearer exception message that distinguishes between these cases will prevent similar headaches that I have gone through.
    
    Author: Andrew Or <andrewor14@gmail.com>
    
    Closes #2067 from andrewor14/python-exception and squashes the following commits:
    
    742f823 [Andrew Or] Further clarify warning messages
    e96a7a0 [Andrew Or] Distinguish between unexpected output and no output at all