Skip to content
  • Dhruve Ashar's avatar
    298c4ae8
    [SPARK-16018][SHUFFLE] Shade netty to load shuffle jar in Nodemanger · 298c4ae8
    Dhruve Ashar authored
    ## What changes were proposed in this pull request?
    Shade the netty.io namespace so that we can use it in shuffle independent of the dependencies being pulled by hadoop jars.
    
    ## How was this patch tested?
    Ran a decent job involving shuffle write/read and tested the new spark-x-yarn-shuffle jar. After shading netty.io namespace, the nodemanager loads and shuffle job completes successfully.
    
    Author: Dhruve Ashar <dhruveashar@gmail.com>
    
    Closes #13739 from dhruve/bug/SPARK-16018.
    298c4ae8
    [SPARK-16018][SHUFFLE] Shade netty to load shuffle jar in Nodemanger
    Dhruve Ashar authored
    ## What changes were proposed in this pull request?
    Shade the netty.io namespace so that we can use it in shuffle independent of the dependencies being pulled by hadoop jars.
    
    ## How was this patch tested?
    Ran a decent job involving shuffle write/read and tested the new spark-x-yarn-shuffle jar. After shading netty.io namespace, the nodemanager loads and shuffle job completes successfully.
    
    Author: Dhruve Ashar <dhruveashar@gmail.com>
    
    Closes #13739 from dhruve/bug/SPARK-16018.
Loading