Skip to content
Snippets Groups Projects
  • Josh Rosen's avatar
    41afa165
    [SPARK-8652] [PYSPARK] Check return value for all uses of doctest.testmod() · 41afa165
    Josh Rosen authored
    This patch addresses a critical issue in the PySpark tests:
    
    Several of our Python modules' `__main__` methods call `doctest.testmod()` in order to run doctests but forget to check and handle its return value. As a result, some PySpark test failures can go unnoticed because they will not fail the build.
    
    Fortunately, there was only one test failure which was masked by this bug: a `pyspark.profiler` doctest was failing due to changes in RDD pipelining.
    
    Author: Josh Rosen <joshrosen@databricks.com>
    
    Closes #7032 from JoshRosen/testmod-fix and squashes the following commits:
    
    60dbdc0 [Josh Rosen] Account for int vs. long formatting change in Python 3
    8b8d80a [Josh Rosen] Fix failing test.
    e6423f9 [Josh Rosen] Check return code for all uses of doctest.testmod().
    41afa165
    History
    [SPARK-8652] [PYSPARK] Check return value for all uses of doctest.testmod()
    Josh Rosen authored
    This patch addresses a critical issue in the PySpark tests:
    
    Several of our Python modules' `__main__` methods call `doctest.testmod()` in order to run doctests but forget to check and handle its return value. As a result, some PySpark test failures can go unnoticed because they will not fail the build.
    
    Fortunately, there was only one test failure which was masked by this bug: a `pyspark.profiler` doctest was failing due to changes in RDD pipelining.
    
    Author: Josh Rosen <joshrosen@databricks.com>
    
    Closes #7032 from JoshRosen/testmod-fix and squashes the following commits:
    
    60dbdc0 [Josh Rosen] Account for int vs. long formatting change in Python 3
    8b8d80a [Josh Rosen] Fix failing test.
    e6423f9 [Josh Rosen] Check return code for all uses of doctest.testmod().