Skip to content
Snippets Groups Projects
  • Marcelo Vanzin's avatar
    37a5e272
    [SPARK-4809] Rework Guava library shading. · 37a5e272
    Marcelo Vanzin authored
    The current way of shading Guava is a little problematic. Code that
    depends on "spark-core" does not see the transitive dependency, yet
    classes in "spark-core" actually depend on Guava. So it's a little
    tricky to run unit tests that use spark-core classes, since you need
    a compatible version of Guava in your dependencies when running the
    tests. This can become a little tricky, and is kind of a bad user
    experience.
    
    This change modifies the way Guava is shaded so that it's applied
    uniformly across the Spark build. This means Guava is shaded inside
    spark-core itself, so that the dependency issues above are solved.
    Aside from that, all Spark sub-modules have their Guava references
    relocated, so that they refer to the relocated classes now packaged
    inside spark-core. Before, this was only done by the time the assembly
    was built, so projects that did not end up inside the assembly (such
    as streaming backends) could still reference the original location
    of Guava classes.
    
    The Guava classes are added to the "first" artifact Spark generates
    (network-common), so that all downstream modules have the needed
    classes available. Since "network-common" is a dependency of spark-core,
    all Spark apps should get the relocated classes automatically.
    
    Author: Marcelo Vanzin <vanzin@cloudera.com>
    
    Closes #3658 from vanzin/SPARK-4809 and squashes the following commits:
    
    3c93e42 [Marcelo Vanzin] Shade Guava in the network-common artifact.
    5d69ec9 [Marcelo Vanzin] Merge branch 'master' into SPARK-4809
    b3104fc [Marcelo Vanzin] Add comment.
    941848f [Marcelo Vanzin] Merge branch 'master' into SPARK-4809
    f78c48a [Marcelo Vanzin] Merge branch 'master' into SPARK-4809
    8053dd4 [Marcelo Vanzin] Merge branch 'master' into SPARK-4809
    107d7da [Marcelo Vanzin] Add fix for SPARK-5052 (PR #3874).
    40b8723 [Marcelo Vanzin] Merge branch 'master' into SPARK-4809
    4a4ed42 [Marcelo Vanzin] [SPARK-4809] Rework Guava library shading.
    37a5e272
    History
    [SPARK-4809] Rework Guava library shading.
    Marcelo Vanzin authored
    The current way of shading Guava is a little problematic. Code that
    depends on "spark-core" does not see the transitive dependency, yet
    classes in "spark-core" actually depend on Guava. So it's a little
    tricky to run unit tests that use spark-core classes, since you need
    a compatible version of Guava in your dependencies when running the
    tests. This can become a little tricky, and is kind of a bad user
    experience.
    
    This change modifies the way Guava is shaded so that it's applied
    uniformly across the Spark build. This means Guava is shaded inside
    spark-core itself, so that the dependency issues above are solved.
    Aside from that, all Spark sub-modules have their Guava references
    relocated, so that they refer to the relocated classes now packaged
    inside spark-core. Before, this was only done by the time the assembly
    was built, so projects that did not end up inside the assembly (such
    as streaming backends) could still reference the original location
    of Guava classes.
    
    The Guava classes are added to the "first" artifact Spark generates
    (network-common), so that all downstream modules have the needed
    classes available. Since "network-common" is a dependency of spark-core,
    all Spark apps should get the relocated classes automatically.
    
    Author: Marcelo Vanzin <vanzin@cloudera.com>
    
    Closes #3658 from vanzin/SPARK-4809 and squashes the following commits:
    
    3c93e42 [Marcelo Vanzin] Shade Guava in the network-common artifact.
    5d69ec9 [Marcelo Vanzin] Merge branch 'master' into SPARK-4809
    b3104fc [Marcelo Vanzin] Add comment.
    941848f [Marcelo Vanzin] Merge branch 'master' into SPARK-4809
    f78c48a [Marcelo Vanzin] Merge branch 'master' into SPARK-4809
    8053dd4 [Marcelo Vanzin] Merge branch 'master' into SPARK-4809
    107d7da [Marcelo Vanzin] Add fix for SPARK-5052 (PR #3874).
    40b8723 [Marcelo Vanzin] Merge branch 'master' into SPARK-4809
    4a4ed42 [Marcelo Vanzin] [SPARK-4809] Rework Guava library shading.