Skip to content
Snippets Groups Projects
  • Marcelo Vanzin's avatar
    af258382
    [SPARK-3217] Add Guava to classpath when SPARK_PREPEND_CLASSES is set. · af258382
    Marcelo Vanzin authored
    When that option is used, the compiled classes from the build directory
    are prepended to the classpath. Now that we avoid packaging Guava, that
    means we have classes referencing the original Guava location in the app's
    classpath, so errors happen.
    
    For that case, add Guava manually to the classpath.
    
    Note: if Spark is compiled with "-Phadoop-provided", it's tricky to
    make things work with SPARK_PREPEND_CLASSES, because you need to add
    the Hadoop classpath using SPARK_CLASSPATH and that means the older
    Hadoop Guava overrides the newer one Spark needs. So someone using
    SPARK_PREPEND_CLASSES needs to remember to not use that profile.
    
    Author: Marcelo Vanzin <vanzin@cloudera.com>
    
    Closes #2141 from vanzin/SPARK-3217 and squashes the following commits:
    
    b967324 [Marcelo Vanzin] [SPARK-3217] Add Guava to classpath when SPARK_PREPEND_CLASSES is set.
    af258382
    History
    [SPARK-3217] Add Guava to classpath when SPARK_PREPEND_CLASSES is set.
    Marcelo Vanzin authored
    When that option is used, the compiled classes from the build directory
    are prepended to the classpath. Now that we avoid packaging Guava, that
    means we have classes referencing the original Guava location in the app's
    classpath, so errors happen.
    
    For that case, add Guava manually to the classpath.
    
    Note: if Spark is compiled with "-Phadoop-provided", it's tricky to
    make things work with SPARK_PREPEND_CLASSES, because you need to add
    the Hadoop classpath using SPARK_CLASSPATH and that means the older
    Hadoop Guava overrides the newer one Spark needs. So someone using
    SPARK_PREPEND_CLASSES needs to remember to not use that profile.
    
    Author: Marcelo Vanzin <vanzin@cloudera.com>
    
    Closes #2141 from vanzin/SPARK-3217 and squashes the following commits:
    
    b967324 [Marcelo Vanzin] [SPARK-3217] Add Guava to classpath when SPARK_PREPEND_CLASSES is set.
pom.xml 12.95 KiB