Skip to content
Snippets Groups Projects
  • Andrew Or's avatar
    a2262cdb
    [SPARK-1735] Add the missing special profiles to make-distribution.sh · a2262cdb
    Andrew Or authored
    73b0cbcc introduced a few special profiles that are not covered in the `make-distribution.sh`. This affects hadoop versions 2.2.x, 2.3.x, and 2.4.x. Without these special profiles, a java version error for protobufs is thrown at run time.
    
    I took the opportunity to rewrite the way we construct the maven command. Previously, the only hadoop version that triggered the `yarn-alpha` profile was 0.23.x, which was inconsistent with the [docs](https://github.com/apache/spark/blob/master/docs/building-with-maven.md). This is now generalized to hadoop versions from 0.23.x to 2.1.x.
    
    Author: Andrew Or <andrewor14@gmail.com>
    
    Closes #660 from andrewor14/hadoop-distribution and squashes the following commits:
    
    6740126 [Andrew Or] Generalize the yarn profile to hadoop versions 2.2+
    88f192d [Andrew Or] Add the required special profiles to make-distribution.sh
    a2262cdb
    History
    [SPARK-1735] Add the missing special profiles to make-distribution.sh
    Andrew Or authored
    73b0cbcc introduced a few special profiles that are not covered in the `make-distribution.sh`. This affects hadoop versions 2.2.x, 2.3.x, and 2.4.x. Without these special profiles, a java version error for protobufs is thrown at run time.
    
    I took the opportunity to rewrite the way we construct the maven command. Previously, the only hadoop version that triggered the `yarn-alpha` profile was 0.23.x, which was inconsistent with the [docs](https://github.com/apache/spark/blob/master/docs/building-with-maven.md). This is now generalized to hadoop versions from 0.23.x to 2.1.x.
    
    Author: Andrew Or <andrewor14@gmail.com>
    
    Closes #660 from andrewor14/hadoop-distribution and squashes the following commits:
    
    6740126 [Andrew Or] Generalize the yarn profile to hadoop versions 2.2+
    88f192d [Andrew Or] Add the required special profiles to make-distribution.sh