Skip to content
Snippets Groups Projects
  • Josh Rosen's avatar
    9cc6ff9c
    Do not launch JavaGateways on workers (SPARK-674). · 9cc6ff9c
    Josh Rosen authored
    The problem was that the gateway was being initialized whenever the
    pyspark.context module was loaded.  The fix uses lazy initialization
    that occurs only when SparkContext instances are actually constructed.
    
    I also made the gateway and jvm variables private.
    
    This change results in ~3-4x performance improvement when running the
    PySpark unit tests.
    9cc6ff9c
    History
    Do not launch JavaGateways on workers (SPARK-674).
    Josh Rosen authored
    The problem was that the gateway was being initialized whenever the
    pyspark.context module was loaded.  The fix uses lazy initialization
    that occurs only when SparkContext instances are actually constructed.
    
    I also made the gateway and jvm variables private.
    
    This change results in ~3-4x performance improvement when running the
    PySpark unit tests.